Email this article July - 2005
Structure, Content, Delivery, Service, and Outcomes: Quality e-Learning in higher education
Abstract
This paper addresses the need for quality e-Learning experiences. We used the Demand-Driven Learning Model (MacDonald, Stodel, Farres, Breithaupt, and Gabriel, 2001) to evaluate an online Masters in Education course. Multiple data collection methods were used to understand the experiences of stakeholders in this case study: the learners, design team, and facilitators. We found that all five dimensions of the model (structure, content, delivery, service, and outcomes) must work in concert to implement a quality e-Learning course. Key themes include evolving learner needs, the search for connection, becoming an able e-participant, valued interactions, social construction of content, integration of delivery partners, and mindful weighing of benefits and trade-offs. By sharing insights into what is needed to design and deliver an e-Learning experience, our findings add to the growing knowledge of online learning. Using this model to evaluate perceptions of quality by key stakeholders has led to insights and recommendations on the Demand Driven Learning Model itself which may be useful for researchers in this area and strengthen the model.
Quality has been defined in terms of the design of the e-Learning experience, the contextualized experience of learners, and evidence of learning outcomes (Carr and Carr, 2000; Jung 2000; Salmon, 2000). Quality and design of e-Learning courses, however, are sometimes compromised in an “ . . . effort to simply get something up and running” in response to pressing consumer demands (Dick, 1996, p. 59). Educators and researchers have voiced concern over the lack of rigorous evaluation studies of e-Learning programs (e.g., Arbaugh, 2000; Howell, Saba, Lindsay, and Williams, 2004; Lockyer, Patterson, and Harper, 1999; Robinson, 2001). McGorry (2003) adds, “although the number of courses being delivered via the Internet is increasing rapidly, our knowledge of what makes these courses effective learning experiences is limited” (p. 160). In an economic environment marked by intensive competition between educational institutions, producing and ensuring quality e-Learning programs will be a competitive advantage to attract learners to post secondary institutions (Daniel, 1996; Duderstadt, 1999).
In this study we used a credible model, the Demand-Driven Learning Model (DDLM), (MacDonald, Stodel, Farres, Breithaupt, and Gabriel, 2001) and its companion evaluation tool (MacDonald, Breithaupt, Stodel, Farres, and Gabriel, 2002) to design and evaluate an online course. Several data collection methods were used to understand the experiences of key stakeholders in this case study: learners, design team, and facilitators. In addition to adding to the growing knowledge of online learning, our findings highlight additional elements that could be incorporated into the DDLM to further refine the model.
Review of the Literature
The availability of well-designed, effectively implemented, and efficiently delivered online courses is essential in order to satisfy the unique needs of growing numbers of adult learners (Daugherty and Funke, 1998; Palloff and Pratt, 2001). Some academics, however, regard the “potential benefits [of e-Learning] as utopian claims and unlikely to hold true in practice” (Furnell, Evans, and Bailey, 2000, p. 283). Still others caution that because the pedagogical soundness of e-Learning has not yet been fully investigated, there is not enough empirical evidence to support claims of its effectiveness (McElhinney and Nasseh, 1999; Noble, 2002; Reeves and Reeves, 1997; Speck, 2000).
The literature reveals a distressing gap between the use of technology and sound pedagogical models (Khan 1997; Salmon, 2000; Willis 2000). Several researchers have written about the need for quality standards to ensure the academic integrity of e-Learning programs (Benson, 2003; Carstens and Worsfold, 2000; DeBard and Guidera, 2000; Salmon, 2000; Speck, 2000). Defining these quality standards, however, can be challenging. In a qualitative study of 13 participants from six different stakeholder groups engaged in developing an online degree program, Benson (2003) found that although everyone wanted quality courses, stakeholders brought different definitions of quality, which impacted the planning process and shaped the learning experiences.
Because of these challenges, evaluating and assuring quality in e-Learning programs has become a critical issue. Not only is evaluation critical for program improvement and long-term success (Rovai, 2003), Marquardt and Kearsley (1998) suggest “evaluation is particularly important in the context of technology use because it [technology] is highly susceptible to fads and marketplace trends” (p. 246). Robinson (2001) reports that in her three-year international study of distance learning initiatives, evaluation efforts were limited, largely due to lack of time and expertise.
McGorry (2003) emphasizes that “theory-driven empirical research is necessary so that criteria for developing effective Internet-based programs are established”(p. 160). The DDLM proposes five inter-related dimensions that in concert, create a high-quality e-Learning experience: structure, content, delivery, service, and outcomes. In this case study, each of these dimensions is applied to a new e-Learning course and explored in-depth. Initial research demonstrates that the DDLM (see Figure 1) provides a psychometrically sound evaluation tool that can be used to assess the quality of e-Learning against reputable guidelines (Breithaupt and MacDonald, 2003).
Figure 1. The Demand Driven Learning Model (DDLM) (MacDonald, C. J., Stodel, E., Farres, L., Breithaupt, K., and Gabriel, M. A. (2001). The demand driven learning model: A framework for Web-based learning. The Internet and Higher Education, 1(4), 9 – 30.
In the DDLM, the structure of an e-Learning course provides the necessary foundation for quality content, delivery, and service. Structure includes anticipating learner needs, using appropriate pedagogical strategies, creating a positive learning environment, and conducting regular learner evaluations. The delivery of an e-Learning course includes usability, interactivity, and tools. The content of an e-Learning course should be comprehensive, authentic, and researched. Service includes resources, administration and technical support, accessibility, and responsiveness. Finally, the outcomes of an e-Learning course should include lower costs for the learner and employer, personal advantages, and achievement of learning outcomes.
The purpose of this study was to address the need for quality assurance in e-Learning. Specifically we wanted to:
- Use a research-based and tested learning model and evaluation tool (the
DDLM) to evaluate an e-Learning course in a university setting
- Synthesize the experiences and needs of key stakeholders
- Provide recommendations to make the DDLM a more comprehensive quality standard.
Research Design
Merriam (2001) comments that the case study is useful for studying learning innovations. Creswell (1998) defines a case study as an exploration of a “bounded system . . . through detailed, in-depth data collection involving multiple sources of information, rich in context” (p. 61). The researcher’s goal is to understand the complexities of the case (Stake, 2000). As we studied one group of people in a unique setting the results cannot be generalized (Wolcott, 1990). Nevertheless, Merriam advises that by providing detail, readers can determine themselves “whether the research setting sufficiently resembles their own situation to warrant adopting the same practices” (p. 222).
The unit of analysis in this case study was EDU 5199, the Synthesis Seminar, the last of 10 courses required to obtain an MEd degree at the University of Ottawa. Because this was the first course in the program to be offered online, learners had completed all other coursework in a F2F format. As the culminating course, participants reflect on their development and demonstrate that they can apply their learning to topical issues in their professional domain.
Participants in this case study were the learners, design team, and facilitators. The 19 learners in the 13-week course were primarily working adults. Thomas (1993) refers to adult education as “any form of education participated in by persons regarded as adults in the society to which they belong” (p. 25). In this study, we refer to the typical MEd student as an adult learner – those learners with at least one previous university degree, many who have family and community responsibilities, and the majority working in full-time jobs. The motivation for designing the course online was to meet the needs of these busy working learners. Attrition was low; 20 learners registered and only one dropped out in the third week reporting demands from her job and family as her reason.
The course design team was comprised of three members. Colla, a professor with 24 years of university level experience, has been researching, publishing, and supervising graduate students in the e-Learning field for several years. This was her first hands-on online teaching experience. The teaching assistant (TA), Terrie Lynn, was completing her MA. She works as a learning consultant, and has designed and delivered courses in several e-Learning media. The course developer, Sylvie, is an experienced WebCT instructional designer. The facilitation team consisted of Colla, assisted by Terrie Lynn. Salmon (2000) refers to an e-moderator as “the champion who makes the learning come alive” by enabling ‘meaning making’ rather than content transmission (p. 11). Accordingly, Colla referred to herself as the e-moderator, rather than professor or instructor.
Data Collection and Analysis
The literature suggests a number of strategies to increase the trustworthiness of a study. Specifically addressing case studies, Stake (1995) recommends member checks and triangulation. Interview participants were encouraged to revise and elaborate on their interview transcripts. Because each data collection strategy “makes the world visible in a different way” (Denzin and Lincoln, 2000, p. 4), relying on multiple strategies helped construct a multifaceted understanding of this e-Learning event. Data was collected in three ways: interviews, analysis of course transcripts and other course documents, and online survey instruments, including the DDLM online survey.
To increase credibility, the semi-structured interviews were conducted by a research assistant who was not involved in the course design or delivery. These interviews allowed us to follow-up on themes that emerged from the course generated data. For example, one re-occurring theme was that learners felt they missed F2F contact. Interviews provided the opportunity to ask learners to explain in greater detail what it was about F2F contact that they missed; we were able to ensure a more complete picture of the learners’ perspectives emerged.
The three members of the design team as well as seven learners were interviewed six weeks after the course finished. Purposeful sampling was used to select learners who varied in age, gender, depth and breadth of course activity, previous e-Learning experience, course achievements, and concentration (the MEd program includes four concentrations: Administration, Teaching and Learning, Counselling, and Second Language Teaching). Eight interviews were conducted in-person, one by telephone, and one by email. Oral interviews were audio taped and transcribed.
Course documentation comprised the second data source and included postings in the e-discussions, emails sent and received by the e-moderator and TA, and course participation data. Other documents available for analysis included course design files, emails between the design team members, and course documents.
The third data source was the online surveys. The course was designed to include several evaluation mechanisms, creating what Levy (2003) refers to as a “close connection between learning and research activities” (p. 101). A required activity in the course, we had 100 percent completion of the three temperature checks. Completed in weeks 3 and 6, two of the temperature checks were short Likert-scale online surveys with a few open-ended questions. Although we could determine if a learner had completed the survey, responses were anonymous. The third temperature check was a reflective posting in the week 9 discussion group. In addition to the temperature checks, the DDLM companion survey (MacDonald et al., 2002) was adapted to align with this course. Learners completed this online survey, called the e-Valuation, during the final week of the course. Designed to evaluate the five dimensions of quality outlined in the model (content, delivery, service, structure, and outcomes), it included 41 six-point Likert questions and five open-ended questions. Data pulled from these online surveys is identified in the text as follows: tc1, tc2, or tc3 refers to temperature check 1, 2, or 3; e-V refers to the final e-Valuation.
Merriam (2001) describes data analysis as the process of meaning making. The first grouping of the data was created as we re-read the data and started to see connections. We tried to find commonalties and concepts that would unite many divergent perspectives. Not surprising, the data did not slot neatly into categories. Themes evolved as we continued to work with the data, and we returned to the data to search for positive and negative evidence. For example, once community surfaced as a potential theme, we re-analyzed the interview transcripts to understand how the learners described the sense of community they felt. This level of analysis resulted in the creation of unifying concepts that more accurately encompassed the meaning emerging from the data. It also allowed us to search for connections between and among participants. Seven themes eventually emerged that attempt to capture the range of experiences and meanings of the participants.
As the research focus of this study was to use the DDLM as an evaluation tool, the model did frame some of our data analysis work. This framework, however, did not restrict the themes that emerged. Schwandt (2000) comments that we invent models to make sense of experience and we “continually test and modify these constructions in light of new experience” (p. 197). The five dimensions in the DDLM were broad enough to enable a grouping of all the themes that emerged. In some instances the data highlighted aspects of e-Learning not reflected in the current DDLM model; these insights are presented in the Conclusion as recommendations for future work on the model.
Wolcott (1994) suggests that analysis reveals “key factors and relationships between them” (p. 10) and by discussing these relationships a case study presents “the potential for understanding something beyond it” (p. 33). Direct quotations are used throughout to present the voices of the participants, who have been given pseudonyms. As the two authors played dual roles as researchers and participants in this online learning experience, we have used our real names throughout this document, guided by Charmaz and Mitchell’s (1997) argument for audible authorship. They state that voice “clarifies the researcher’s place in, and experience of, that action” (p. 208).
The Context
Work began seven months before the course was to be offered. At that time, the only e-Learning technology supported by the university was WebCT, which established the online environment. Guided by the DDLM, design decisions linked this e-Learning technology with sound pedagogical principles.
The Synthesis Seminar, the final course in the MEd program, requires learners to reflect on their professional development throughout the program and demonstrate that they can apply new learnings by writing a 25-page research paper. There was no prescribed course content to be delivered. The content was primarily constructed by the learners as they wrote their paper and shared insights in the discussion groups and consequently, knowledge built throughout this learning experience was directly related to professional and personal interests. The only content provided by course developers were brief online documents designed to support learners in this constructivist learning environment; some provided guidance for writing sections of the paper while others provided insights into successful e-Learning (e.g., About Netiquette, Tips for WebCT Discussions). Even though this was an asynchronous course a sense of timing was established by creating a course chunked into 13 weekly modules, each ending Wednesday at midnight. The Road Map provided structure by identifying the learning events for each module.
As mandated by the University, this pass/ fail course was based on assessment of the research paper. To help learners produce the best possible paper, a series of assignments guided them step-by-step through the process of writing an academically rigorous research paper. Learners were required to: (a) participate in online activities; (b) develop a conceptual framework; (c) create an outline of the paper; (d) submit a first draft of the paper; (e) develop an online poster presentation, and (f) produce a final paper. All assignments were evaluated and extensive feedback was provided on each. Although only the final paper was given a grade, the cumulative assignments facilitated formative assessment and enabled learners to build their final papers in stages as they continually refined their research and arguments. To help define “online participation,” several online documents were available (e.g., Building a Learning Community, Hints for Successful e-Learning). As there was no mark assigned per se to online participation, we relied on several strategies to encourage it: (a) online participation was included in the list of course requirements to emphasize its importance; (b) we also tried to design online forums that learners would find useful and that had the potential to engage a diverse group of people in a meaningful dialogue that would enhance their research paper; and (c) the focus of each discussion group was highlighted in the course outline and Colla posted frequent reminders and encouragement.
A constructivist philosophy underlies the DDLM. While there are different viewpoints among constructivists (Barab and Duffy, 2000), Levy (2003) suggests they all acknowledge that learning is active, situated, and social. She adds that “in practice this leads to a commitment to participatory and dialogic approaches to learning design and facilitation, including . . . participation in learning communities” (p. 93). Consequently the following strategies were implemented to facilitate a sense of community: (a) learners were grouped into triads and expected to provide constructive feedback and support to their triad members; (b) dialogue in the discussion groups was emphasized; and (c) learners were able to meet the facilitation team and other learners during an initial three-hour F2F class.
The course design included an optional three-hour long F2F orientation session in the first week, designed to help ease learners into their first e-Learning experience and ensure a successful online experience. While the intent was to enable learners to participate online at anytime and from anyplace, in order to facilitate community and orient them to WebCT, this F2F session became a course design element. In this respect, offering one of the thirteen classes F2F reflects a blended approach. It is possible, however, for learners to successfully participate in the online course without attending this F2F session.
The triads were designed to provide feedback and support mechanisms. Learners self-selected triad members during the initial F2F session. With one learner dropping out and learners grouping themselves by interest areas there were five triads of three members, and two triads of two members. Triad members were expected to give each other feedback on various course assignments. Learners used special forms to provide each other feedback and attached the feedback forms they have received when they submitted their assignments. The feedback they were given by their triad members and the quality of the feedback they provided to their triad members was assessed.
Eight one-week discussion groups were strategically placed throughout the course. In the discussions, learners were asked to share: 1) aspects of their paper; 2) opinions and insights; 3) their journey as e-learners; and 4) how their research related to challenges in their professional work. The facilitation team was responsive and active in the discussion groups. Learners also used WebCT email and chat tools to dialogue.
To help learners orient themselves to the e-Learning environment we created E-Venture, a series of activities learners completed in the first week to familiarize themselves with WebCT. One of these activities was to create and post an e-Page, which included a short introduction and image (optional). We also tried to anticipate challenges learners might face in writing the research paper. This led to the design of tools such as the online American Psychological Association (APA) self-assessment to help learners figure out the nuances of academic referencing. Strategically placed at three-week intervals, three temperature checks gave learners an opportunity to provide learner feedback, enabling us to respond to learner needs as the course progressed. Toll free technical support for password, connectivity, and hardware problems was provided by the University help desk, although it was not a 24 x 7 support service.
Findings
Learners had their own perceptions of quality and used other learning experiences as benchmarks. One learner comments: “I have taken one other course via WebCT. The organization of EDU5199 was exceptional and much higher than my expectations!” (e-V) Another shares, “I really felt supported to rise towards the next step. Ironically, I felt more support than I have in other courses, even though there was no face-to-face contact” (e-V). Learners believed that the design and facilitation teams demonstrated a commitment to quality and continuous improvement. J. explains:
I think they were very open to suggestions of how this could be improved and what was working well . . . Asking people to comment in this fashion is a very good sign that they’re trying to take where it is and make it easier for people . . . It looks easy to the outside person like me – all the professor had to do was start this up and then step back. But that took a lot of work at the front end and there’s obviously a commitment too at the tail end, going back and trying to refine and change(Interview).
Learners reported that this online course was a positive experience. For example: “This has been a challenging and rewarding experience. In my opinion, this is how e-Learning should be. We had relevant work to do, great organization and support, and almost no technical glitches” (Posted by C. on Sunday, December 01, 2002: 13:50 p.m.). For one learner however, the stress caused by having to take a course online was perceived as too great:
I learned a lot from it, but to me the stress that I suffered because of having to go through that [online course] . . . It just didn’t work for me . . . And people I have talked to loved the course, but it did not meet my needs (M., interview).
While the standards in this online course were extremely high, learners felt they were set up to succeed and all passed. In her opinion, Colla found that the quality of the learners’ work was high in comparison to other graduate courses she had taught. For example, her perception was that the quality of the first drafts of the learners’ conceptual frameworks were strong. Several factors might help explain this perception. First, Colla developed an online document that included a step-by-step process and examples of conceptual frameworks. In F2F courses, such detailed upfront documentation rarely occurs as the professor can rely on his/ her knowledge and experience to react to learners concerns and questions. However, perhaps learners in this course benefited from information organized and presented in this written readily accessible format. Due to the online nature of the course, the design team also organized learners into learning triads to provide additional support. A second reason the quality of the assignments was higher may be due to learners providing feedback to others in their triad prior to submitting assignments.
The DDLM model provides a framework for exploring the findings in more depth. Seven themes emerged from the data: a) evolving learning needs; b) search for connection; c) becoming an able e-participant; d) valued interactions; e) social construction of the course content; f) integration of support by delivery partners; and g) mindful weighing of benefits, drawbacks, and trade-offs. These themes will be discussed under the five components of the DDLM: structure, content, delivery, service, and outcomes.
Structure
This section discusses two structure-related themes that emerged: evolving learning needs and a search for connection.
Evolving learning needs
In the fall term, EDU5199 was the first course offered completely online and was available only in an online format. In the interviews, all but one learner expressed feeling angry or anxious. P., however, was happy when he heard the news: “There was no stress at all. If I could have picked to do every course online I would have done that” (Interview). Although learners reported being told they could take the course F2F if they waited until the next term, for most this was not a viable option. Given a choice at that point, most would have elected to take this course F2F. Moreover, learners were made aware that the course was being offered online three weeks prior to the first class, which seemed to add to their anxiety level.
Although the majority of learners reported feeling forced to take the course online, when it was over, most saw it as a feasible, and even enjoyable, learning experience that they would try again. It seems learners’ reactions to their first e-Learning experience reflect both the perceived attractiveness of the learning option as well as “do-ability.” Most learners experienced a shift in their attitude, in part because of the structure and design. K. explains:
During the first class I felt ‘Wow this is too much. I don’t know if I’m going to make it’ . . . Once you were online though, I found that that was completely gone. Everything was set in place and if you had any inquiries you could contact someone. So the design of the whole online program was excellent, really easy to use (Interview).
Our findings suggest that many learners experienced an initial doubting phase when they wondered if they could do this e-Learning course. Learners’ perceptions of their own e-Learning competence and attitude toward this new learning experience seemed to influence their level of confidence which in turn influenced how they engaged in the course. D. and J. admitted to limited technical competence, yet looked on this e-Learning experience as an adventure and with a few early successes felt a sense of progress, and eventually competence. The data suggests that perceptions may shift during the experience. J. shares, “I went from nervousness about it to feeling a bit better, to feeling quite good about it towards the end (Interview).
In contrast, M. ended up feeling defeated. Although her self-reported technology skills were similar to D. and J., the technology seemed to create insurmountable barriers despite the level of effort she put in: “I tried so hard and even with trying so hard there were just too many obstacles” (Interview). As the course progressed her confidence decreased while her stress increased. Her initial feeling of “horror” at learning this would be an online course never abated. She admits that she would not rush into taking another course online.
Studying learners’ perceptions of starting an online course, Conrad (2002) also found that some learners felt fear and anxiety; others were eager and excited. Students also experienced “initial discomfort with the online medium” (p. 215). Our data supports her finding that courses with “good beginnings” are characterized by organization and social ambience (p. 215). Consistent with Palloff and Pratt’s (2001) assertion, it seems some learners struggle, while others adapt successfully and easily to the online setting.
Search for connection
Design efforts focused on fostering a sense of community. Learners reported that the triads were invaluable and did foster a sense of working together towards a common purpose. There were differing views, however, on the value of the e-discussions and three challenges related to fostering community via the e-discussion venue emerged. The first challenge of learning to manage the volume of postings is logistical in nature. Several learners felt guilty about not reading and responding to all the postings. All shared the strategies they used to cope with the volume.
Bridging the diversity of professional backgrounds (teachers, school administrators, and counselors) and research topics presented a larger challenge to fostering a sense of community. For some learners, a lack of interest or expertise in the topics areas explored by other learners created a barrier to dialogue. Learners were uncertain how to respond and some doubted that their comments would be perceived as credible. Several learners, however, not only managed to bridge this apparent gap, they valued the diversity of perspectives and enjoyed the exchanges with others. They drew on their own experiences to connect with the issues that others were exploring.
Third, the search for connection with others in the course seemed to influence the way people participated and the resulting feeling of community. Some learners found it more difficult to establish a social context in this online environment. A few learners attributed this to lack of non-verbal cues, while others felt that more F2F time together at the beginning would have provided a stronger foundation. Many learners met their triad members in-person or communicated synchronously by telephone. Some learners shared that they needed to work harder to find and create the exchanges that seem to occur so naturally in a F2F session. J. elaborates:
I like to be able to sit back and listen to the exchange of ideas among people . . . Initially I didn’t find that in this particular setting, because you have to go looking for it and it’s in the print form as opposed to engaging in a conversation and dialoguing. That was different for me . . . At the beginning it was more stressful to create that same exchange of ideas that takes place in a classroom (Interview).
One course element that seemed to help participants create a connection with others was the e-Pages. Many of the participants, including the e-moderator, referred back to the e-Pages throughout the entire 13 weeks. This type of bond made the learning experience more real for some participants in this study. Other researchers have also found that bringing participants together F2F before the online component and posting each person’s digital picture helps to build an online community (Bichelmeyer, Misanchuk and Malopinsky, 2001; Song, Singleton, Hill and Koh, 2004). In a recent study which examined the perspectives and experiences of 22 learners in an asynchronous online course, Vonderwell (2003) found that students were not comfortable interacting with others they did not know. By turning to synchronous communication modes in their triads, learners in this study instinctively discovered a strategy consistent with Macdonald’s (2003) findings of online collaborative learning: a F2F meeting, teleconference, or online chat helped learners establish a working relationship before working online.
Macdonald (2003) also found that individuals contribute to the group process in several ways including group moderation, responding encouragingly to others, or locating information. In exploring how learners attempted to span diverse interests, our study goes one step further by highlighting the importance of a common language in building connections, and ultimately a community. Wenger (1998) suggests that coherence in a community comes with mutual engagement, joint enterprise, and a shared repertoire (words, ways of doing things, stories). While learners in this study were dispersed across four professional domains, three ‘shared repertoires’ or common languages seemed to emerge and learners used these to connect with others: emotional support, technology, and the process of e-Learning.
Even though participation in the discussion groups was not evaluated per se, it was a course requirement and our findings support assertions in the literature that some learners may not feel obligated to participate and some will be more enthusiastic participants than others (Macdonald, 2003; Vonderwell, 2003). Learners reported that they participated in the online discussion groups largely because this was a course requirement. More spontaneous and genuine participation, however, occurred when they found them useful, were interested in what others were doing, felt they had something to contribute, wanted to provide support to others, and in order to maximize the return on their tuition investment. There was also a sense in this study that learners participated in the spirit of reciprocity: ‘give in order to receive.’ Synthesizing the findings of several case studies, Hammond (2000) concluded that learners who focused on the opportunities provided by the discussion groups became “communicative learners” while “learners who focused on the constraints perceived a high threshold to cross before they could join in” (p. 256). Our findings illustrate the diverse factors that learners weigh when deciding on their level of participation and the value they derive from a joint learning endeavor.
Some learners in this study seemed to bring F2F expectations with them. In her personal inquiry into e-Learning, Mann (2003) shares that she tried to “engage with others through written communication as if I was in a F2F conversation” (p. 118). She argues that, “A whole new communication process has to be learned. It is not simply a process of shifting from speaking and listening to reading and writing” (p. 119). While most learners felt there was a sense of community in this e-Learning experience, the findings highlight how ability to connect and dialogue with others influences if, and how, a sense of community emerges. It seems that being able to dialogue meaningfully in the online environment is a pre-requisite for building community. Although community was planned into the design of this course, more active facilitation and purposeful focus on this dimension was likely required to help it take root and blossom.
Delivery
Two themes illustrate how delivery influenced the quality of this online learning experience: becoming an able e-participant and valued interactions.
Becoming an able e-participant
For some learners a source of anxiety was perceived technology incompetence and an inability to confirm whether they had the necessary technology skills to be successful in this e-Learning experience. Learners were not the only one who doubted their e-Learning skills. Colla shares: “[The learners] think they were intimidated or scared? I was scared too. I’m not a technology person and never ever pretend to be and to take this on you have to be crazy. So I was scared too” (Interview).
Despite technology skill building activities, such as e-Venture in the first week, for many learners there were two learning curves in this course: the technology and the research paper. Comments in the first temperature check (week 3) reveal that computer-related struggles seemed to eat up time and energy often at the expense of the paper. Even as learners expressed frustration at coping with the challenges of a new learning environment, for many there was a realization that the technology was becoming less of a barrier: “It was difficult in the beginning . . . but by the end of the course I felt very comfortable with everything . . . After 3-4 weeks, I was right on track and felt I could do it very competently and with speed” (D., interview).
Three weeks into the course, much of the stress over the technology and consequently the stress in the class appeared to subside. Interestingly, learning to use PowerPoint to create a conceptual framework, as well as basic word processing skills, seemed to create more obstacles than using WebCT. For some, this challenge eclipsed all others. By week 6, midway through the course, there was a positive shift in the learners’ confidence and ability to use WebCT (see Table 1). Now 15 participants compared to 9 three weeks earlier, were feeling comfortable with the online environment.
Table 1. Growing Confidence in the e-Learning Environment
The data suggests that not everyone slides easily into this e-Learning medium. There is a sense of adjustment and struggle and the biggest hurdles varied from learner to learner. For some, it was dealing with hardware, software, and networks glitches; for others, adjusting to the lack of F2F contact. Some learners seemed to cope better than others. All became more cognizant of their computer skills. Learners also shared their emerging sense of self as an e-learner. With this increased self-awareness they developed strategies. Some focused on time management and although they knew what they needed to do, acknowledged that this online course ended up fitting haphazardly around the ebb and flow of other priorities. Several consciously bettered their technology skills.
For some participants there was a sense of progression. They acknowledged that next time they would be more proficient and therefore able to participate more meaningfully, taking full advantage of the online course elements. Budding e-Learning ability is referenced in the literature. Mann (2003) suggests that becoming an able e-participant takes time and reflects: “Potential was not realized, for in my mind I was still learning the ropes afforded by this new medium, was still translating it out of my previous experience, rather than experiencing it in its generative newness” (p. 121).
Most learners in our study did not face any major technology problems and shared that the e-Learning interface (WebCT) was easy to use and relatively problem-free. Most learners also indicated that they would consider taking other courses online. The findings support the literature that suggests that learner satisfaction with Internet-based courses will influence their perspectives of the usefulness of the technology for e-Learning and whether they take subsequent courses in this format (McGorry, 2003; Song et al., 2004). As Rheingold (2000) states, the way a technology will be used depends on how well the “first people who are using it succeed or fail in applying it to our lives” (p. xxvii).
Valued interactions
Over the 13 weeks the survey, data suggests that the number of learners who believed they were engaged in this learning experience varied from 14 to 16. Despite this level of engagement, the data also reveals that half of the learners were either not sure or did not think they had enough interactivity with others (see Table 2).
Table 2. Perceptions of engagement and interactivity
Perceptions of interactivity were in part influenced by the virtual nature of the exchanges. One learner comments: “I am feeling slightly isolated without the social aspects of learning, in spite of the fact that my triad has met several times in the chat rooms” (tc1). As the course progressed, the desire for F2F contact became more pronounced by some learners. Barely mentioned in the first temperature check, by week six, F2F meetings came up repeatedly as one suggested improvement. Perhaps it is the synchronous nature of F2F contact that is missed, as this comment suggests: “I do miss the F2F contact when I need some feedback or clarification right away” (tc2).
In the interviews, learners revealed several factors that influenced their interactions in the virtual space. Some learners did not participate because of divergent interests. How others might perceive them and uncertainty about the context in which others might be reading their comments also emerged, suggesting that perhaps the foundation of trust and support required to nurture a community was not strong enough for some learners. Other learners referred to time constraints and being selective in their interactions. JS. admitted that he was a lurker, reading more than he posted, and that his level of interaction was a conscious decision: “I didn’t gain as much as I might have . . . I think I might have let the group down in that respect. But you also have to weigh your life demands as well” (Interview).
The data suggests that creating vibrant and open online interaction is a challenge given time constraints and a sense of uncertainty about how input would be regarded by others. The interview data, however, also suggests that some learners found the degree of online interaction required was good. Perhaps learners’ perceptions of the level of interactivity were influenced by how much they participated in the online discussions and how much they valued this type of interaction. CK was an international student and valued the text-based nature of course: “I can follow this course without losing track. Since all texts were written, I could go back to some points that I need to reread” (Interview). However, she confined her interactions to her triad members, and primarily with the one person she knew from a previous F2F class. In her interview she shared: “As for the other classmates, unfortunately, I had no relationship with anybody.” Even though she adds: “I am sure the discussion groups facilitates interactivity among us,” she admits that the “lack of communication with the other learners” was the least rewarding aspect of the course.
These findings raise questions: What strategies are necessary to facilitate an e-Learning course that encourages learners to value online interactions? Might the asynchronous text-based element of the medium be problematic? Would anything other than F2F contact and interactivity be sufficient? J. comments: “I can think of courses where a big part of it has been the professor and the interaction. I don’t know that with Colla. I didn’t get enough time to work with her F2F” (Interview). Despite the email dialogue between the e-moderator and J., comments they shared in the discussion groups, and the extensive amount of personal feedback she provided to him on five assignments, he is still not sure he really interacted with her. Differing perceptions of interaction and valued interaction point to the need for further research in this area.
McGorry (2003) acknowledges that creating interactivity, crucial to long-term interest in e-Learning, is one of the greatest challenges. McLoughlin and Oliver (1998) argue that the quality of an e-Learning experience is related to the social climate and the “opportunities created for interaction and exploratory talk between participants” (p. 134). The design team in this case study was focused on creating an interactive learning experience, a key component of the DDLM. Interactivity as outlined in the DDLM includes the interaction between the learner and other learners, the facilitators, and the content. The findings in this study illustrate the challenges in building an interactive e-Learning experience that appeals to everyone. Learners appeared to bring their own beliefs about the interactions they value with the content, themselves, the instructor, the interface, and other learners. Rather than being an all-encompassing concept, interactivity has multiple facets. The data suggests we need to be more precise in facilitating interactivity, which supports Thompson’s (2003) finding in her study of online teaching and learning, that attention needs to be paid to the type of interactions promoted in the virtual classroom.
Content
The content-related theme emerging from the data is the social construction of the course content. Most course content was constructed by learners as they crafted their paper, exchanged feedback with triad members and the e-moderator, and shared ideas in the online discussion groups. Learners seemed to value the flexible nature of the content, as this comment illustrates: “It was a strength of the course that all of the course content was either directly related to writing the synthesis paper or our personal e-Learning. No time was wasted on irrelevant examples” (e-V). According to the e-Valuation, the content seemed to meet most learners’ needs, have sufficient breadth and depth, and enabled learners to synthesize what they had learned throughout the MEd program (see Table 3).
Table 3. Evaluation of the course content
This study provides a rich illustration of a constructivist approach to content. Learners in this study did not absorb knowledge; they constructed it; a key tenet of constructivism (Jonassen, 1991; Jonassen, 1994; Phillips, 1995; Vygotsky, 1978). There is also evidence that the learners’ experience was consistent with Fenwick’s (2000) assertion that constructivism is characterized by an experience in which learners “construct, through reflection, a personal understanding of relevant structures of meaning derived from his/her action in the world” (p. 248).
This study highlights the challenges of this constructivist approach. The small amount of prescribed course content, coupled with the personal and diverse nature of the topics explored in the papers, made it challenging for learners to find a common focal point as a group. The literature offers some insight. Hung and Chen (2001) suggest commonality (shared interests and problems) contributes to a vibrant community. Did the participants in EDU5199 see a valid reason to work together to address a challenge? To some degree, a sense of commonality did emerge in the triads as learners were invested in the feedback process on assignments. Within the large group, however, this sense of common purpose was not well-defined. Our findings indicate that if a key structural component of an online course is community, then the content, as well as the learning process and assessment criteria, need to align with this type of learning experience.
Service
Integration of support by delivery partners emerged as a theme that fits within the service dimension of the DDLM. Delivery partners in this e-Learning experience included the university technical support and faculty administrative staff. Not all learners interacted with the support staff. The interview data reveals that learners with higher levels of technology skills (i.e., P. and J.) did not seek technical support. Several learners, however, did express concern about the quality of technical support. The data also suggests that most learners built and relied upon a personal support network, which included the facilitation team, each other (triad members and the large group), and “experts” at home and work.
Learners repeatedly commented that they valued and benefited from the support offered by the e-moderator, TA, and their learning triads. Responsiveness of the facilitation team, demonstrated by quick turn around on questions, concerted efforts to help solve problems, and comprehensive feedback to learners on their assignments illustrates a commitment to service. Table 4 highlights the learners’ perceptions of this service.
Table 4. Perceptions of service
The interviews revealed that several learners with marginal technical skills successfully built support networks. Both D. and J. relied on IT personnel in their schools as well as calling on others in the course. Neither seemed hesitant to ask for help. M. however, did not seem to have a strong personal support network to help her through her numerous software challenges. Furthermore, she was not comfortable asking for help and struggled throughout the 13 weeks:
My problem was probably that I was too reluctant to let on to [Colla] that I was having difficulties . . . I wouldn’t contact her because I just thought I should be able to figure it out by myself. She was very supportive but I didn’t reach out . . . I work with special needs kids . . . I did not reach out, which is typical of a learner in difficulty(Interview).
The data also reveals that the design and facilitation teams built an e-Learning support network as well, which included SY., who provided pedagogical assistance as well technical expertise and access to a resource network within the University. Findings also highlight the negotiation of resources that went on to build this support network. Terrie Lynn observes:
I remember our initial meeting with the e-Learning center and then discovering these kinds of things take money and is there any money to do this? . . . It’s not just a matter of creating an online course. This whole infrastructure has to also be created to support whatever you do(Interview).
Colla acknowledges the effort involved in providing the degree of support she did for the learners, admitting that she would be online first thing in the morning and continue to check email and e-postings throughout the day, often “10 times a day.” Terrie Lynn provides insight into how the facilitation team juggled their service priorities:
Colla was so busy marking the papers, and learners have commented so positively on the feedback she’s given them, on how detailed it was . . . And that comes at a price. You can’t be busy doing that, you know 50 hours a week plus reading all the discussion groups plus troubleshooting people’s technical problems plus answering APA questions. I don’t know how people who don’t have a TA manage. You would maybe compromise by settling for a less interactive experience. But if there had been more time [it would have been good to] follow up [individually] on people who were low participants . . . and just say, “I’m just checking in to see how things are going”(Interview).
While the University technical support and faculty administrative staff are clearly part of an integrated delivery model of e-Learning, it is questionable how these groups see their role. Although they were integrated to some extent into an overall service framework, they are not necessarily prepared or committed to an intensive e-Learning delivery model. This study highlights that an integrated approach to service is an essential element of a quality e-Learning experience. Even with quality course design and delivery, shortfalls in the service aspect of a quality learning experience, which often fall outside the direct influence of the facilitators, can have a significant impact on learners’ experiences.
In this case study, there was no faculty e-Learning strategy or departmental requirements or incentives to create e-Learning courses. The drive to create an online version of EDU5199 was primarily due to the determination of one professor and her ability to marshal the necessary resources and willingness to take a risk by trying something new. Our findings support the literature that suggests many of these early successes are linked with the enthusiasm and capability of the individual professor (McLachlan-Smith and Gunn, 2001; Robinson, 2001). Robinson observes that strategies used by e-Learning innovators include “working around intransigent procedures, making informal arrangements with selected administrators on a personal goodwill basis,” and absorbing some of the administrative work themselves (p. 19). McLachlan-Smith and Gunn comment that learning innovation at the University of Auckland has been driven primarily by early adopters, who, against all odds, created e-Learning courses within the traditional infrastructure.
The findings in this study highlight the challenges faced by the design team in trying to access resources to develop and deliver an e-Learning experience. Observations by learners of the inconsistencies between an online course, on the one hand, and F2F registration systems and technical support that is not prepared for the 24 x 7 nature of e-Learning, on the other hand, support suggestions that institutions must develop e-Learning policies that will “maintain course integrity and quality and also foster innovation in the ‘virtual classroom’” (McGorry, 2003, p. 160). Alexander (2001), however, observes that most e-Learning activity is at the course level and only a few institutions recognize that “successful e-Learning occurs within a complex system, composed of many inter-related parts” (p. 241). She concludes that to meet its challenges, higher education e-Learning initiatives will need to move beyond teaching strategies; her framework includes the support and development mechanisms within the entire university context.
Outcomes
One theme emerged to illustrate the outcomes dimension of an e-Learning experience: a mindful weighing of benefits, drawbacks, and trade-offs. Missing F2F contact was an issue that became more pronounced by some learners as the course progressed. It seems some learners missed the fluidity and familiarity of verbal exchanges. But even as some learners commented they wanted more F2F meetings, by week 6, the convenience of this e-Learning experience was now being recognized:
I miss F2F contact, but know I can seek that out for myself with my triad members. The only problem is that now that I am in a habit of NOT travelling to campus on a set night each week, it is more of a chore now to set up F2F meetings (tc2).
Learners commented favourably on the ease of submitting assignments, fewer transportation issues, cost savings, and increased flexibility. Persistence of the online dialogue was also an advantage to many. For some learners there was no sense of a trade-off in taking this course online rather than F2F. They were enthused about this course and enjoyed the learning experience wholeheartedly, as this comment illustrates:
I loved the online format. It gave me the flexibility I needed, but the course was structured enough to support me in writing my paper. I’m an independent learner and was able to access support whenever required from my triad, the TA, or the instructor (e-V).
In the interviews several learners commented on the efficiency of this e-Learning experience. P. notes that the convenience aspect would not have been a benefit if the course had not been so well organized and efficient:
I think if anything I gained more from this course because you aren’t saddled down in those three hour classes where a lot of them you don’t get anything out of . . . It was all in the organization. If it was haphazardly organized, we’d have been in big trouble (Interview).
For others, the convenience, novelty, and opportunity to experience e-Learning firsthand made this experience palatable. There was a sense of giving up the familiar comforts of the F2F classroom routine, but also a feeling that they gained much more than they gave up. For example: “I am thoroughly enjoying this experience. The convenience far outweighs any drawbacks. The technology works when I need it and I’m not running into any technical difficulties” (tc2). But for a few, this e-Learning experience did not work for them as the trade-offs and compromises were too great. M. shares: “My confidence has been challenged by the online learning format. There is a definite clash between my learning style and e-Learning” (Posted on Friday, November 01, 2002, 23:56).
Many of the learners reported that they were applying what they learned in their work context, consistent with the expectations of the DDLM. Furthermore, participants were now more versed in the e-Learning medium, valued the firsthand experience, and were now recommending online learning to colleagues. For many learners, a significant benefit was their personal growth. In the final e-Valuation, 100 percent of the learners indicated they achieved personal or professional growth. There was a sense of adventure. Perceived as a friendly push to take a risk and try something different, learners felt a sense of accomplishment at the end of the course. For many, this online course was a significant journey, as this posting from J. indicates:
Early in September . . . as I looked ahead, with trepidation to e-Learning, I wondered if perhaps I had taken on more technology than I could handle . . . I am grateful for the feedback that my fellow classmates and instructors provided . . . I gained a great deal from the various courses along the way and e-Learning was the perfect cap to the program(December 02, 2002 20:35).
As the designer and facilitator of this course, Colla also felt a sense of accomplishment and shares, “it’s rewarding to know that although it was so much time and effort it wasn’t just for this course, materials can be used in other courses” (Interview).
Conclusions
This study highlights the challenges of offering a quality online experience in an organization still shaping an approach to e-Learning. Our findings suggest that the innovation and implementation of sustainable high quality e-Learning courses must be part of a systematic integration of technology into the learning processes of the university; a challenge in a traditional F2F program. As learners shared the challenges and successes of participating in this e-Learning experience, insights into the characteristics of an immersive robust e-Learning culture emerged. In such a setting, learners choose between different learning media. They are better prepared to participate successfully in a challenging high quality e-Learning experience because they are introduced to the use of learning technologies and software applications progressively throughout their program, building the requisite technology and learning skills. Along with the option to register online and access to other university support services online, learners have 24 x 7 access to reliable technical help. Such an e-Learning culture is characterized by a true integration of delivery partners: (a) technical support and administrative staff; (b) e-Learning design and facilitation teams; and (c) decision makers who determine resource allocation and media. As this study illustrates, quality e-Learning comes with a cost: significant investments in time and energy. For sustainable quality e-Learning programs, this investment must be acknowledged and supported. McLachlan-Smith and Gunn (2001) state that securing resources is “an ongoing concern in a university not used to the ‘front end’ resource implications” of e-Learning (p. 43).
Designing a quality e-Learning course is a complex process. Perceptions of quality in this course seemed to be strongly linked to: (a) a fit between the content and design of this e-Learning experience and the learners’ needs, wants, and perceived competence; (b) ability to accomplish meaningful outcomes, including enhanced computer literacy – which Dewey (1938/1970) refers to as collateral learning; (c) overall learning efficiency; and (d) the quality of feedback from the e-moderator.
Furthermore, this study highlights that all five dimensions of the DDLM (structure, content, delivery, service, and outcomes) are intertwined throughout the design, delivery, and evaluation of an e-Learning course. It is the collective impact of these dimensions that leads to a cohesive e-Learning experience. Rovai (2003) suggests that since online programs consist of several components, such as the “e-Learning software, academic and technical support, presentation of content, and interaction, evaluators must recognize that all components of the program must work together in an efficient manner if the entire system is to be effective” (p. 113). The findings of this study go one step further by suggesting that there is more to designing and implementing a quality e-Learning experience that just understanding the key components. Process emerges as a key factor that links these components into a responsive and relevant situational learning experience. Given resource requirements and the degree of collaboration required to deliver a quality e-Learning event, a professional approach to instructional design and project management is essential.
Built on learning research, theory, and practice, the DDLM served as a framework to guide the design and delivery of a quality e-Learning experience. Evaluating the success of this online course based on the five dimensions outlined in the DDLM provided constructive feedback to the design and facilitation teams that can be used to improve future deliveries. Using the DDLM to evaluate perceptions of quality by key stakeholders has also led to insights on the model itself, which may be useful for researchers in this area.
First, although the model emphasizes the importance of collaboration and interactivity, it could be more robust by integrating the concept of community. Even though creating a supportive learning community was challenging in this course, it is still a relevant element in a quality learning experience and deserves further exploration and experimentation. Faced with similar findings, Song et al. (2004) argue that because community continues to be an issue, more research is needed. It seems there is a need to work with learners to assist them with establishing community or at least, feelings of connection in online contexts. As O’Connor (1998) notes, researchers are grappling with the tension between the collective and the individual. Perhaps socio-cultural theories and perspectives will provide additional theoretical insights.
Second, the structure dimension of the DDLM emphasizes the importance of understanding learners’ needs and motivations. Findings from this study suggest that more attention must be paid to the sense of confidence and perceived level of competence that seems to influence the decisions participants make about if, and how, they will engage in an e-Learning course. Technology competence, however, is not only the ability to master the interface. Ability to use supporting applications (i.e., Word or PowerPoint), troubleshoot system problems, and self-assess e-Learning requisite skills is important. More precise articulation and separation of the Web interface from the supporting software applications in the survey questions (i.e., the e-Valuation) would provide more accurate data.
Lastly, MacDonald et al. (2001) suggest that the DDLM is “not a list of required ingredients of success; rather it is a recipe whereby any WBL program can succeed” (p. 27). To continue to improve the recipe, the model could feature more specific guidelines on how to design and deliver a quality e-Learning experience; this may enable more deliberate application of strategies that lead to a quality e-Learning experience. For example, this study reflected the ongoing nature of the design process; it didn’t end once the course started. Moreover, design is intertwined with facilitation strategies. As suggested by the participants in this study, designing an e-Learning course that lends itself to rapid re-design as learners’ needs become better articulated is characteristic of a quality e-Learning experience. Facilitation strategies that enable instructors to adjust the course design in situ may be useful to incorporate into the model.
The amount of anxiety M. experienced throughout the course is both disturbing and perplexing. It is disturbing as M. choose to keep her frustrations to herself. Had this become more apparent to the instructors, it is possible that additional support could have been provided which might have relieved some of her stress. M’s experience invites further research into what prevents learners from reaching out for help when they find themselves in an online course that does not seem to suit their learning style and whether it is possible to turnaround a negative e-Learning experience. Alternatively, perhaps we have to resolve our thinking to the fact that e-Learning may not for everyone. This study also suggests other areas for research, such as: (a) including other stakeholders as study participants (e.g., administrative and e-Learning support staff as well as university leaders) to wrestle with how to create an integrated network of delivery partners; (b) exploring what kinds of interactivity are really valued by the learners so that they are able to connect with others and engage in more effective learning experiences; and (c) investigating how to move stakeholders (learners, designers, and facilitators) past the first hurdles quickly so that they feel confident and competent.
The flexibility of the DDLM to guide both the design and evaluation of an e-Learning experience illustrates the dynamic intersections between theory and best practices. As Vogel (2000) explains, “[when] theoretical frameworks inform actions, and actions modify theories so that future actions grow out of what we have learned by experience and reflection, the entire system is energized” (p. 25). As researchers continue to build frameworks based on theory and reflection of practical experiences, the resulting insights will enable all stakeholders in an online experience to make more informed decisions to positively impact the quality of e-Learning. This study is one step in helping to build a broad base of theoretical knowledge informed by practical experiences.
References
Alexander, S. (2001). E-learning developments and experiences. Education + Training, 43(4/5), 240 – 248.
Arbaugh, J. B. (2000). Virtual classroom characteristics and student satisfaction with Internet-based MBA courses. Journal of Management Education, 24(1), 32 – 54.
Barab, S. A., and Duffy, T. M. (2000). From practice fields to communities of practice. In D. H. Jonassen and S. M. Land (Eds.) Theoretical Foundations of Learning Environments (25-54). Mahwah, NJ.: Lawrence Erlbaum.
Benson, A. (2003). Dimensions of quality in online degree programs. The American Journal of Distance Education, 17(3), 145 – 159.
Bichelmeyer, B. A., Misanchuk, M., and Malopinsky, L. (2001). Adapting a Master’s Degree Course to the Web: A case analysis. The Quarterly Review of Distance Education, 2(1), 49 – 58.
Breithaupt, K., and MacDonald, C. J. (2003). Quality Standards for e-Learning: Cross validation study of the Demand Driven Learning Model (DDLM), Testing International, 13(1), 8 – 12.
Carr, C. S., and Carr, A. M. (2000). Instructional Design in Distance Education (IDDE): A web-based performance support system for educators and designers. The Quarterly Review of Distance Education, 1(4), 317 – 325.
Carstens, R. W., and Worsfold, V. L. (2000). Epilogue: A cautionary note about online classrooms. In R. E. Weiss, D. S. Knowlton, and B.W. Speck (Eds.) Principles of Effective Teaching in the Online Classroom, No. 84 (83-87). San Francisco: Jossey-Bass.
Charmaz, K., and Mitchell Jr., R. G. (1997). The Myth of Silent Authorship: Self, substance, and style in ethnographic writing. In R. Hertz (Ed.) Reflexivity and voice (193-215). Thousand Oaks, CA.: Sage Publications.
Conrad, D. L. (2002). Engagement, Excitement, Anxiety, and Fear: Learners’ experiences of starting an online course. The American Journal of Distance Education, 16(4), 205 – 226.
Creswell, J. W. (1998). Qualitative Inquiry and Research Design: Choosing among five traditions. Thousand Oaks, CA.: Sage.
Daniel, J. S. (1996). Mega-Universities and Knowledge Media: Technology strategies for higher education. London: Kogan Page.
Daugherty, M., and Funke, B. L. (1998). University faculty and student perceptions of web-based instruction. Journal of Distance Education, 13(1), 21 – 39.
DeBard, R., and Guidera, S. (2000). Adapting asynchronous communication to meet the seven principles of effective teaching. Journal of Educational Technology Systems, 28(3), 219 – 230.
Denzin, N. K., and Lincoln, Y. S. (2000). The discipline and practice of qualitative research. In N. K. Denzin and Y. S. Lincoln (Eds.) Handbook of qualitative research (1-28). Thousand Oaks, CA.: Sage.
Dewey, J. (1970). Experience and Education. New York: Collier-MacMillan Canada. (Original work published in 1938).
Dick, W. (1996). The Dick and Carey Model: Will it survive the decade? Educational Technology Research and Development, 44(3), 55 – 63.
Duderstadt, J. J. (1999). Can colleges and universities survive in the information age? In R. N. Katz (Ed.) Dancing with the devil: Information technology and the new competition in higher education (1-25). San Francisco: Jossey-Bass.
Fenwick, T. J. (2000). Expanding Conceptions of Experiential Learning: A review of the five contemporary perspectives on cognition. Adult Education Quarterly, 50(4), 243 – 272.
Furnell, S., Evans, M., and Bailey, P. (2000). The Promise of Online Distance Learning: Addressing academic and institutional concerns. The Quarterly Review of Distance Education, 1(4), 281 – 291.
Hammond, M. (2000). Communication Within Online Forums: The opportunities, the constraints and the value of a communicative approach. Computers and Education, 35(4), 251 – 262.
Howell, S. L., Saba, F., Lindsay, N. K., and Williams, P. B. (2004). Seven Strategies for Enabling Faculty Success in Distance Education. The Internet and Higher Education, 7(1), 33 – 49.
Hung, D. W. L., and Chen, D. T. (2001). Situated Cognition, Vygotskian Thought and Learning from the Communities of Practice Perspective: Implications for the design of Web-based e-learning. Education Media International, 38(1), 3 – 12.
Jonassen, D. H. (1991). Objectivism versus Constructivism: Do we need a new philosophical paradigm? Educational Technology Research and Development, 39(3), 5 – 14.
Jonassen, D. H. (1994). Thinking Technology: Toward a constructivist design model. Educational Technology, 34(4), 34 – 37.
Jung, I. (2000). Technology Innovations and the Development of Distance Education: Korean experience. Open Learning, 15(3), 217 – 231.
Khan, B. H. (1997). Web-based Instruction (WBI): What is it and why is it? In B. H. Khan (Ed.) Web-based Instruction (5-18). Englewood Cliffs, NJ: Educational Technology Publications.
Khan, B. H. (2001). A Framework for Web-based Learning. In B.H. Khan (Ed.) Web-based Training (75-98). Englewood Cliffs, NJ: Educational Technology Publications.
Levy, P. (2003). A Methodological Framework for Practice-based Research in Networked Learning. Instructional Science, 31(1-2), 87 – 109.
Lockyer, L., Patterson, J., and Harper, B. (1999). Measuring Effectiveness of Health Education in a Web-based Learning Environment: A preliminary report. Higher Education Research and Development, 18(2), 233 – 246.
MacDonald, C. J., Breithaupt, K., Stodel, E., Farres, L., and Gabriel, M. A. (2002). Evaluation of Web-based Educational Programs: A pilot study of the Demand-Driven Learning Model. International Journal of Testing , 2(1), 35 – 61.
MacDonald, C. J., Stodel, E., Farres, L., Breithaupt, K., and Gabriel, M. A. (2001). The Demand Driven Learning Model: A framework for Web-based learning. The Internet and Higher Education, 1(4), 9 – 30.
Macdonald, J. (2003). Assessing Online Collaborative Learning: Process and product. Computers and Education, 40(4), 377 – 391.
Mann, S. J. (2003). A personal Inquiry into an experience of adult learning online. Instructional Science, 31(1-2), 111 – 125.
Marquardt, M. J., and Kearsley, G. (1998). Technology-Based Learning. Boca Raton, FL.: St. Lucie Press.
McElhinney, J. H., and Nasseh, B. (1999). Technical and pedagogical challenges faced by faculty and students in computer-based distance education in higher education in Indiana. Journal of Educational Technology Systems, 27(4), 349 – 359.
McGorry, S. Y. (2003). Measuring quality in online programs. The Internet and Higher Education, 6(2), 159 – 177.
McLachlan-Smith, C., and Gunn, C. (2001). Promoting innovation and change in a ‘traditional’ university setting. In F. Lockwood, and A. Gooley (Eds.) Innovation in Open and Distance Learning: Successful development of online and Web-based learning (38-50). London: Kogan Page.
McLoughlin, C., and Oliver, R. (1998). Maximising the language and learning link in computer learning environments. British Journal of Educational Technology, 29(2), 125 – 136.
Merriam, S. B. (2001). Qualitative Research and Case Study Applications in Education San Francisco: Jossey-Bass.
Noble, D. F. (2002). Digital Diploma Mills: The automation of higher education. New York: Monthly Review Press.
O’Connor, M. C. (1998). Can we trace the “efficacy of social constructivism”? Review of Research in Education, 23, 25 – 71.
Palloff, R. M., and Pratt, K. (2001). Lessons from the Cyberspace Classroom: The realities of online teaching. San Francisco: Jossey-Bass.
Phillips, D. C. (1995). The Good, the Bad, and the Ugly: The many faces of constructivism. Educational Researcher, 24(7), 5 – 12.
Reeves, T. C., and Reeves, P. M. (1997). Effective dimensions of interactive learning on the World Wide Web. In B. H. Khan (Ed.) Web-based Instruction (59-66). Englewood Cliffs, NJ.: Educational Technology Publications.
Rheingold, H. (2000). The Virtual Community: Homesteading on the electronic frontier. Cambridge, MA.: The MIT Press.
Robinson, B. (2001). Innovation in Open and Distance Learning: Some lessons from experience and research. In F. Lockwood, and A. Gooley (Eds.) Innovation in Open and Distance Learning: Successful development of online and Web-based learning. (15-26). London: Kogan Page.
Rovai, A. P. (2003). A practical framework for evaluating online distance education programs. The Internet and Higher Education, 6(2), 109 – 124.
Salmon, G. (2000). e-Moderating: The key to teaching and learning online. London: Kogan Page.
Schwandt, T. A. (2000). Three Epistemological Stances for Qualitative Inquiry: Interpretivism, hermeneutics, and social constructionism. In N. K. Denzin and Y. S. Lincoln (Eds.) Handbook of qualitative research (189-213). Thousand Oaks, CA.: Sage.
Song, L., Singleton, E. S., Hill, J. R., and Koh, M. H. (2004). Improving Online Learning: Student perceptions of useful and challenging characteristics. The Internet and Higher Education, 7(1), 59 – 70.
Speck, B. W. (2000). The Academy, Online Classes, and the Breach in Ethics. In R. E. Weiss, D. S. Knowlton, and B. W. Speck (Eds.) Principles of Effective Teaching in the Online Classroom, No. 84 (73-81). San Francisco: Jossey-Bass.
Stake, R. E. (1995). The Art of Case Study Research. Thousand Oaks, CA.: Sage.
Stake, R. E. (2000). Case Studies. In N. K. Denzin and Y. S. Lincoln (Eds.) Handbook of Qualitative Research (435-454). Thousand Oaks, CA.: Sage.
Thomas, A. M. (1993). The New World of Continuing Education. In T. Barer-Stein, and J.A Draper (Eds.) The Craft of Teaching Adults, (21-38). Toronto: Culture Concepts.
Thompson, T. L. (2003). Even a Virtual Synchronous Classroom has Walls: There’s more to collective meaning making than the technology. Unpublished master’s thesis. University of Ottawa, Canada.
Vogel, L. J. (2000). Reckoning with the Spiritual Lives of Adult Educators. In L. M. English and M. A. Gillen (Eds.) Addressing the Spiritual Dimensions of Adult Learning: What educators can do, No. 85 (17-27). San Francisco: Jossey-Bass.
Vonderwell, S. (2003). An Examination of Asynchronous Communication Experiences and Perspectives of Students in an Online Course: A case study. The Internet and Higher Education, 6(1), 77 – 90.
Vygotsky, L. S. (1978). Mind in Society: The development of higher psychological processes. In M. Cole, V. J. Steiner, S. Scribner, and E. Souberman (Eds.) Cambridge, MA.: Harvard University Press.
Wenger, E. (1998). Communities of Practice: Learning, meaning, and identity. Cambridge, UK.: Cambridge University Press.
Willis, J. (2000). The Maturing of Constructivist Instructional Design: Some basic principles that can guide practice. Educational Technology, 40(1), 5 – 16.
Wolcott, H. F. (1990). Writing up Qualitative Research. Newbury Park, CA.: Sage.
Wolcott, H. F. (1994). Transforming Qualitative Data: Description, analysis, and interpretation. Thousand Oaks, CA.: Sage.
