• Home
• About
• Register
• Current
• Archives
• Announcements
• Resources
• Submissions
• Reviewers
• Conferences

Home > Vol 14, No 5 (2013) > Shattuck

December – 2013

Using a Design-Based Research Study to Identify Principles for Training Instructors to Teach Online

Julie Shattuck¹ and Terry Anderson^2
1Frederick Community College, USA, ²Athabasca University, Canada

Abstract

Within the overall framework of design-based research, this paper reports on a study that focused on evaluating an online training course for online instructors. This intervention was designed as a possible solution to the problem facing many higher education institutions of how to provide quality, accessible training for mostly part-time instructors who are making the transition to teaching online. The research project explored whether the training course had any impact on the participants’ later teaching practice. The major outcome of this research study is the identification of design principles that can be used by other researchers and practitioners designing online instructor training.

Keywords: Online teaching; adjunct faculty; design-based research; faculty training; online teaching professional development

Introduction

Within the overall framework of design-based research (Barab & Squire, 2004; Design-Based Research Collective, 2003; Sandoval & Bell, 2004), this paper reports on a research project that focused on evaluating a training course for online instructors: MarylandOnline’s Certificate for Online Adjunct Teaching (COAT) course. COAT was designed as a possible solution to the problem facing some higher education institutions of how to provide quality, accessible training for part-time instructors who are making the transition to teaching online. COAT is a nine-week, online, instructor-led course that was designed to prepare adjunct faculty to teach their first online course. Although designed specifically for adjunct faculty with no online teaching experience, COAT participants were found to be more diverse than originally planned for with full-time faculty, administrators, and instructors with extensive prior online teaching experience enrolling in the course (Shattuck, 2013). This article reports the findings from a study that explored the impact of the COAT course on the participants’ later teaching practice.

Design-Based Research

Design-based research (DBR), also called design experiments (Brown, 1992), design research (Collins, Joseph, & Bielaczyc, 2004), and educational design research (McKenney & Reeves, 2012), has generated increasing interest among educational researchers in the last decade (Anderson & Shattuck, 2012). Wang and Hannafin (2005) defined DBR as “A systematic but flexible methodology aimed to improve educational practices through iterative analysis, design, development, and implementation, based on collaboration among researchers and practitioners in real-world settings and leading to contextually-sensitive design principles and theories” (pp. 6-7). DBR positions researchers with practitioners as part of a team that works together, usually over an extended period of time, to provide a solution(s) to a practical problem that faces a specific educational context. DBR studies use the term intervention to denote the object, activity, or process that is designed as a possible solution to address the identified problem. McKenney and Reeves (2012) identified intervention as a broad term used “to encompass the different kinds of solutions that are designed” (p. 14); these solutions include educational products, processes, programs, and policies. This study identified the COAT course as the intervention that was developed as a potential solution to the perceived need for better high quality training for online adjunct faculty.

DBR projects can span many years with multiple research cycles that focus on the iterative stages of the project analysis, design, development, implementation, and evaluation phases. In order to clearly explain how the research reported in this paper was positioned within a collaborative, ongoing DBR project, it is helpful to use McKenney and Reeves’ (2012) generic model for design research (GMDR) to provide an outline of the COAT project phases. The GMDR consists of three main phases, analysis and exploration, design and construction, and evaluation and reflection, that lead to the two eventual outputs of increased theoretical understanding and effective intervention maturation. The three phases of analysis/exploration, design/construction, and evaluation/reflection interact with ongoing practice as the intervention is adopted, enacted, and sustained (implementation) in a particular educational setting while information about the intervention is disseminated and diffused to a wider audience.

Using the GMDR to frame the COAT project, this study is situated in phase three: evaluation/reflection. Table 1 briefly outlines how the COAT project aligned with the initial iteration of the three phases of the GMDR with implementation added to the second phase in order to include how the COAT project implemented the first iteration of the training course after a successful pilot.

In DBR, the content, structure, and instructional approaches of an intervention are first identified in the analysis and exploration phase of a design project through a literature review and the input of experts and practitioners. This information is then used to design the first iteration of the intervention. A preliminary literature review is conducted with the purpose of identifying draft design principles that have the potential to address the problem the intervention is being designed to solve. In the COAT project, the draft design guidelines included what content, structure, and instructional approaches might best be used to address the training and experiential needs of adjunct faculty who are making the transition to teaching online.

A preliminary literature review was conducted in fall 2008 to spring 2009 and the results were disseminated (Dubins & Graham, 2009). Findings from that literature review, combined with interviews with key personnel from 17 US training courses for online instructors and a survey of distance learning administrators and faculty trainers at Maryland higher education institutions that offer online courses, informed the development of the first iteration of the COAT course. This course placed instructors as students within a paced, asynchronous online course that was structured to be similar to the type of online courses they may teach in higher education institutions. The results and discussion of the design/construction phase which included a pilot run of the course are published elsewhere (Shattuck, Dubins, & Zilberman, 2011). This paper focuses on the initial evaluation/reflection phase of the COAT project.

DBR has proved to be an effective approach for other research projects focused on the design and evaluation processes of instructor training programs and initiatives (Dede, Ketelhut, Whitehouse, Breit, & McCloskey, 2009). Ketelhut, McCloskey, Dede, Breit, and Whitehouse (2006) highlighted the importance of, and tension between, the dual goals of asking program evaluation questions about the effectiveness of online teacher professional development (oTPD) programs and asking empirical research questions about the impact of oTPD programs, and they identified DBR as a promising approach to address both of these goals. A comparison of DBR to other methodological approaches reinforces the choice of DBR as the appropriate approach for this study.

Collins, Joseph, and Bielaczyc (2004) compared DBR to three types of educational research approaches: laboratory and training studies; ethnographic research; and large-scale studies. They argued that laboratory and training settings do not account for multiple variables, multiple participants’ expertise, and “the messy situations that characterize real life learning” (Collins et al., 2004, p. 20); ethnographic research describes in detail what and why relationships and events occur, but it does not try to change practice; and large-scale studies “do not provide the kind of detailed picture needed to guide the refinement of a design” (p. 21).

Action research has similarities with DBR in terms of collaboration, researchers having multiple roles, and reflection on practice. Typical action research, as opposed to critical action research, positions the teacher-as-researcher conducting “a form of disciplined inquiry, in which a personal attempt is made to understand, improve, and reform practice” (Cohen, Manion, & Morrison, 2007, p. 297). Action research can be collaborative, but the emphasis is on reflective research to inform individual practice at the local level. DBR, in contrast, is always collaborative with a focus on the generation of design principles in an evaluation/reflection phase where reflection “involves active and thoughtful consideration of what has come together in both research and development (including theoretical inputs, empirical findings, and subjective reactions) with the aim of producing new (theoretical) understanding” (McKenney & Reeves, 2012, p. 151). Developing design principles is part of an ongoing DBR process that may eventually lead to theoretical understanding:

The outcomes of design-based research are a set of design principles or guidelines derived empirically and richly described, which can be implemented by others interested in studying similar settings and concerns. While the ultimate objective is the development of theory, this might only occur after long-term engagement and multiple design investigations. (Amiel & Reeves, 2008, p. 35)

One of the goals of the study reported in this paper was to use the detailed data that were collected and analyzed from the evaluation of the first iteration of the COAT course to articulate design principles that are relevant to other distance learning professionals and that are transferable to similar contexts.

Purpose and Research Questions

The purpose of this research study was to evaluate whether the content, structure, and instructional approaches of the COAT course effectively helped prepare higher education instructors to teach online, and, through reflection, to extract design principles that could prove useful for other researchers and practitioners working in the field of online instructor training. The research model used to evaluate the COAT course was Guskey’s (2000) model of five critical levels of professional development evaluation. This study collected and analyzed data focused on evaluating Level 4: participants’ use of new knowledge and skills.

Guskey (2000) argued that most evaluations of professional development occur at Level 1 (initial participant reactions) as it is the easiest level to assess. However, data gathered from the lower levels are not informative for measuring the impact of training on subsequent practice. Guskey highlighted that it is challenging to make a connection between teaching practice and earlier training experiences:

Educators work in complex environments where multiple factors affect their behaviors. Changes in leadership, occurrences in one’s personal life, other learning opportunities, or changes in professional assignment could alter participants’ behaviors and activities quite apart from the influence of professional development. Isolating the professional development experience as the true cause of change in practice is a challenging aspect in any evaluation effort. (2000, p. 187)

This study aimed to meet this challenge through utilizing mixed research methods that included most of the ways Guskey identified for gathering information at Level 4. His methods included: questionnaires; interviews with participants and their supervisors; participant reflections; participant portfolios; direct observations; and video or audio tapes. The research methods used in the doctoral thesis (Shattuck, 2013) in which this study was conducted included a questionnaire, participant reflections in focus groups, analysis of archived online courses, and interviews. This paper reports on the second phase of the doctoral study: online, asynchronous focus groups in which the following research questions were explored:

1. Can a sample of alumni who taught online after completing the COAT course identify any elements (content, structure, instructional approaches, etc.) of the COAT course as being notably important in helping them teach their subsequent online course(s)? If yes, which elements?
2. Do they identify any elements as being unimportant or even misleading in informing their subsequent online teaching practice? If yes, which elements?

Research Paradigm

This study was framed within an interpretivist paradigm which considers that “a primary aim of social science is to understand what people mean and intend by what they say and do and to locate those understandings within the historical, cultural, institutional, and immediate situational contexts that shape them” (Moss et al., 2009, p. 501). Denzin and Lincoln (2011) state that an interpretivist paradigm “assumes a relativist ontology (there are multiple realities), a subjectivist epistemology (knower and respondent co-create understandings), and a naturalistic (in the natural world) set of methodological procedures” (p. 13). Crotty (1998) differentiated between creating understandings, a subjectivist epistemology that sees meaning as being created by individuals, and constructing understandings, a constructionist epistemology that considers that people construct meaning together in relation to their engagement with their human world. This DBR study, with its focus on generating knowledge about a training course from the subsequent activities and reflections of COAT participants, operated within a social constructionist epistemology. According to Koro-Ljungberg, Yendol-Hoppey, Smith, and Hayes (2009), educational researchers working within a social constructionist perspective have multifaceted, participatory roles; research goals to “negotiate and transform the practice” (p. 690); and a view of knowledge as being generated from groups of participants. In addition, this study falls within what Bell (2004) described as a “folk (emic) research orientation that investigates the manifested meaning of an intervention from the point of view of the participants of the research as interpreted through their activity and their accounts” (p. 248). This aligns with DBR’s characteristics of offering practical solutions to real world problems from both the perspectives of the participants and the researchers involved in the design team.

Research Method

This study collected data using online, asynchronous, threaded discussion groups as focus groups to explore the research questions using online discussion boards within a learning management system (LMS). Turney and Pocknee (2005) researched the use of LMS discussion boards for virtual focus groups, and concluded that asynchronous focus groups were theoretically sound because they have the potential to meet Krueger’s (1988) six criteria for making a group a focus group: involving people, being conducted in a series, having relatively homogenous participants who do not know each other, being a method of data collection, collecting qualitative data, and constituting a focused discussion.

Nicholas et al. (2010) summarized the advantages to conducting asynchronous focus groups: convenient access; no time constraints allows participants to reflect which leads to “data depth and richness” (p. 110); participants cannot interrupt each other; emotions can be expressed through emoticons and textual clues; no travel or transcription costs; and face validity is fostered “due to member checking, as participants have continuous access to the data transcript and have ongoing opportunity to reflect on their statements to ensure that meaning is sufficiently captured within the data” (p. 110). Disadvantages include a lack of visual clues, time commitments required of participants, possible technical barriers to participate, and questions about security of data.

An online, asynchronous format was appropriate for the participants of this study for the following reasons. All COAT alumni were accustomed to interacting asynchronously using a discussion board as this was an integral part of the COAT course, and, as such, technical barriers were not expected to be problematic due to the participants’ familiarity with these tools. In addition, the lack of visual clues is something that COAT alumni are used to dealing with in their roles as online learners and instructors. The LMS used to host the focus group discussion boards is a secure site that is password-protected, and participants were given pseudonyms to protect their anonymity. The convenience of interacting asynchronously meant that geographic and time constraints were less likely to impact the feasibility of setting up the groups. Krueger and Casey (2009) suggested inviting asynchronous focus group participants “to spend 15-30 minutes each day for several days as they review the questions and make their responses” (p. 178). By considering the time commitment of about 15-30 minutes a day over a period of several days, participants were able to gauge whether they had the time needed to participate.

Focus Group Participants and Logistics

The purposive sample for the focus groups was derived from the respondents to a questionnaire sent to all participants who had completed one of 11 COAT course sections that ran fall 2010 to spring 2012. All 126 (out of a possible 179) respondents to the online questionnaire were sent an invitation to participate in the focus groups if they had taught online after completing COAT which led to 24 COAT alumni participating in the focus groups.

Participants were offered a choice of five dates for the focus groups. Two dates were not popular which resulted in three separate groups that ran in July, August, and September, 2012: focus group one (FG1), focus group two (FG2), and focus group three (FG3). After signing the informed consent agreement, participants were enrolled in the LMS focus group site using a numeric identifier to maintain anonymity. The researcher’s role in each focus group was to welcome people to the group, facilitate the conversations, and provide summaries of the discussions for member-checking. Each group was held asynchronously using the discussion board feature of the LMS that had hosted all of the COAT courses. Each focus group was held over three days with a separate discussion prompt for each day. FG1 and FG2 had identical prompts, and FG3’s Day One prompt was also identical. However, changes were made to the prompts for the second and third days of FG3 based on the ongoing data analysis results from FG1-2.

Each focus group was opened a few days early for participants to preview. The focus groups were left open for a week after day three finished, so that participants could make any changes or additions to their postings before the data collection period closed and data analysis began. No changes were made, but three participants (one in each focus group) did add a post the day after the third day. The researcher provided summaries for each day’s discussion and a final summary of the whole focus group. Participants were invited to make changes and corrections to these summaries. Only one clarification was suggested, and three participants verified that the summaries had captured what was important from their perspectives.

Focus group participants were not asked to give detailed demographic information about themselves, as this may have compromised their anonymity. However, some participants did disclose personal details in their introductions which included the information that participants held a number of professional roles within education with ten people saying they had worked or were currently working as adjunct faculty, five as administrators, three as full-time faculty, two as instructional technologists, and seven as Kindergarten-12th grade (K-12) instructors. These roles were often held simultaneously with the K-12 instructors, administrators, and technologists working as higher education adjuncts too. Of the nine participants who gave information about how many institutions they were currently employed in, seven people worked in only one institution with two others identifying that they worked at more than one institution simultaneously. The institutions people worked at were varied with ten community college, two university, and one K-12 institutional type identified. Six people had not taught online prior to taking COAT, and 11 people had prior online teaching experience ranging from one course to over ten years.

Additional information about participants included prior experience as online students with 15 (63%) people having taken online courses before participating in COAT and three saying they had no online student experience prior to COAT. Participants also talked about the subjects they taught online which included accounting, art, astronomy, business, child development, communication, computer science, English, health, history, medical assisting, research methods, and statistics.

Analysis of Research Results

This study drew on grounded theory techniques to inform data analysis decisions and followed Saldaña’s (2009) recommendation to approach coding method choices with “pragmatic eclecticism” (p. 47) by letting initial data collection and review occur before deciding on which coding method(s) to use. Data analysis for the three focus groups was ongoing with preliminary analysis beginning after the first focus group ended and further analysis continuing through iterative cycles of initial and focused coding which informed data collection decisions for subsequent focus groups. Constant comparison (Strauss & Corbin, 1998) of new data against previously collected data and existing codes against new codes and emerging categories occurred throughout the process. Detailed notes were kept in reflection blogs and analytic memos during the focus group data collection and analysis period.

Important Elements

All 24 focus group participants identified at least one element of COAT that had positively influenced their subsequent online teaching practice, and their comments were grouped into the category Taking COAT influenced subsequent online teaching practice. This category encompassed 138 quotations which were organized into a code family (see Figure 1). This code family included one category (placed top center in Figure 1); five codes that stated a particular element of taking COAT which influenced later practice or a specified impact on later practice that was attributed to taking COAT (shown with the relationship is a in Figure 1); one code that was seen as contributing to later teaching practice (shown with the relationship contributes to), and 11 subcodes (shown with the relationship is part of). Some of the codes are associated with more than one code and these relationships are depicted with is associated with relationship arrows. The numbers after each code in Figure 1 show the groundedness and density of the code. “Groundedness counts the number of links to quotations; density counts the links to other codes and memos” (Friese, 2012, p. 140).

General comments on COAT’s influence on online practice.

Two codes were populated with general comments that participants made about COAT influencing later teaching practice. Eight quotations referred to participants revisiting COAT content as reference materials for later teaching practice, and 15 quotations were general comments about COAT’s influence on improving or informing later practice. These comments ranged from COAT being seen as an essential part of later online teaching success; to enhancing existing online teaching practice; to COAT being seen as a minor aid to later teaching practice. The majority of quotations (115 out of 138) were more specific about what elements of the COAT course influenced later practice and these quotations were grouped into four codes which are presented next in conjunction with their subcodes.

Experiencing being an online student.

As shown in Figure 1, this code has four subcodes with a combined total of 28 linked quotations that were made by 16 (out of 24) focus group participants. To be included in this code, quotations had to explicitly highlight that the experience of being positioned as a student in the COAT course influenced participants’ subsequent online teaching practice. These experiences could be positive or negative. The 11 quotations directly linked to the code were all positive comments about experiencing life as an online student affecting later online teaching practice. For some participants COAT was their first experience as an online student:

I think participating as a student in the COAT training was of utmost importance. As a novice to online teaching, I really needed to be a student. Plus participating as a student opened my eyes to so many things I would have never even considered if I was just reading about these topics.

Others had taken online courses: “I have been an online student before, but this course helped me to focus on the student experience a little better because I was taking the course as an instructor wanting to provide a better experience for my students.” This concept of benefiting from looking at their own teaching practice through the lens of an online student was mentioned by multiple participants; for example,

I could see what instructional techniques worked well, and which ones did not. Being in the student role made me look at both sides of a situation. Did I as a student learn something from the exercise? Would I as an instructor get valuable feedback from this exercise. If it doesn’t meet both criteria, it must be changed.

This idea of learning from what participants did not like about experiencing life as an online student in COAT was detailed more in two of the four subcodes. Three different participants described feeling frustrated by parts of their COAT experience. For one, the frustration led her to be mindful of students’ possible frustration with the same element in her own courses:

I had not been in the ‘online’ student role for ~7 years when I took the COAT course. I was not familiar with the learning platform used to deliver the COAT course… This unfamiliarity gave me an opportunity to become frustrated and remember many students will experience this when they are in my class.

This idea of negative experiences in the COAT course leading to more awareness of their own students experiencing the same feeling was also apparent in the quotations linked under the subcode Experiencing time commitment; for example,

I also felt that there was an enormous amount of material presented, and was overwhelmed near the end of the class with all of the requirements. I actually went back to my classes again and rethought some of my expectations, juggling assignment positions within the time frame.

Experiencing the COAT course design as a student also led to changes in participants’ later practice with eight quotations related to this idea. For example, one participant highlighted copying design features from COAT that she liked into her own courses:

As a result of taking the COAT course I modified my own courses to incorporate some of the instructional design features from the COAT course… for example, a separate button for “Weekly Course Work,” and separate folders for each week.

Three participants mentioned that COAT introduced them to features of the LMS they had been unaware of which led to them using these features in their own practice.

COAT was designed to give participants the experience of being an online student in a paced, facilitator-led, cohort-based course while learning about teaching in a similarly configured online learning environment. The code Experiencing being an online student and its four subcodes had 28 quotations that specifically highlighted that the way the course was purposefully structured to position participants as online students had an impact on later practice. This code is also associated with three other codes which were populated with quotations that can be seen to relate in part to participants’ experience in the course as students, but primarily highlight different elements of the COAT course, some of which were expected results that aligned with the COAT project’s planned outcomes, and others which were unexpected outcomes. An expected outcome was for participants to reflect on the role of an online instructor.

Reflecting on personal teaching role as online instructors.

This code and its two subcodes encompassed 24 quotations that focused on how taking the COAT course prompted participants to reflect on their role as an instructor and to make changes to both the types of activities they included in their subsequent online courses and their presence in their courses as a result of this reflection. For five participants, a key takeaway from COAT was their role shifting to being a facilitator of the learning process. Other participants discussed how COAT had made them think about their readiness for teaching in the online environment, the importance of their response time to students, how to deal with disruptive students online, and the need for setting up open-ended activities to engage students in the learning process.

A subcode that dug deeper into the idea of student engagement grouped together eight quotations about providing opportunities for peer interaction. Five of these quotations referred to using group work/wikis; for example,

I was personally terrified of the idea of working in a team online, but my COAT experience was exceptional and I vowed to use team experiences whenever possible in my online courses. This has been difficult for some of my students, and some of my student evaluations have been quite negative because of the group work I require, but some of the students share the wonderful experience I got during my group work in COAT and I think the potential for this experience is worth the risk and negativity some will maintain. I feel working in an online group takes the online educational experience to a whole different level, and really represents the best of what online teaching can offer. I only wish it could be a good experience for them all.

This previous quote also demonstrates how this subcode/code can be viewed as being associated with the code Experiencing being an online student, as the experience of doing group work in COAT led to the participant incorporating group work into her subsequent courses which resulted in continued reflection on her teaching practice. Another subcode that is also associated with the experience of being an online student in COAT is Modeling online teaching. This subcode contained three quotations that referred to participants learning from the COAT facilitator modeling good online teaching practice.

Being part of a community of learners.

Just over half of the focus group participants (13 out of 24) identified that a positive benefit of taking COAT was that it provided them with the opportunity to interact with other instructors who were participating in the course, and these interactions played a part in influencing later practice (shown as the relationship contributes to in Figure 1). For example, one participant who had not taught online prior to COAT stated: “During the COAT class I appreciated learning from other students who either had taught online already or who were teaching in real time while taking the COAT class. Their stories and examples were invaluable.” Participants who had already taught online before COAT also found being part of a community of instructors/learners beneficial: “I agree, the COAT course gave me a place to talk to other teachers, to troubleshoot issues with like-folks. We do tend to teach in isolation.” Participants also expressed the wish that the community of learners had survived the end of the course. This desire to interact with other instructors separate from the assigned course curriculum or discussion prompts was demonstrated in the focus groups with the many side discussions that took place between focus group participants that were not directly related to the focus group discussion prompts.

Unimportant or Misleading Elements

Focus group participants were asked to discuss any elements of the COAT course that were unimportant or misleading. Seven people stated that they could not identify any such elements, and six participants identified a variety of elements that they found less helpful or even misleading. There were few commonalities among these elements with individual participants identifying different aspects of the course ranging from student integrity information to the choice of textbook. Other participants made comments that were critical of the COAT course, though not coded as identifying unimportant or misleading elements. For example, constructive criticism in the form of recommendations was made with comments ranging from what was seen as an overwhelming amount of work to improvements to the course design. In addition to providing feedback on the COAT course, participants made suggestions on further training or networking opportunities that the COAT project should consider.

Discussion

In order to reflect on the categories and codes that emerged from the data analysis phase, metaphors were used to abstract the findings to a more conceptual level. Metaphors of immersion in a foreign/alien culture and of COAT being a pebble making ripples in pools of practice led to a further review of relevant literature.

Immersion in an Online Learning Environment

Two-thirds (16 out of 24) of focus group participants identified that their experience of being an online student in the COAT course had influenced their subsequent online teaching practice. Their comments about seeing an online course from a student’s perspective, experiencing frustration with the course navigation, and feeling overwhelmed resonated with a metaphor of how making the transition to online learning and teaching can be compared to living in a foreign country. Being immersed in a new environment and faced with a different culture can first cause frustration, confusion, self-doubt, and fear that can then lead to rethinking what is taken for granted as normal or commonplace behaviors. In a similar way, moving from a campus-based to an online learning environment can be a discombobulating experience that can make instructors question what they feel they know as truths about teaching and learning. Brookfield (1993) argued that “experiencing what it feels like to learn something unfamiliar and difficult is the best way to help teachers empathise with the emotions and feelings of their own learners as they begin to traverse new intellectual terrains” (p. 21). The findings from the COAT alumni research study extends this idea to experiencing what it feels like to learn something in an unfamiliar and difficult learning environment is the best way to help instructors approach their subsequent online teaching from a more student-centered perspective.

This idea of comparing learning how to behave in new academic environments to immersion in new cultures is not original, and examples can be found in the literature of situated learning and communities of practice. For example,

To talk about academic disciplines, professions, or even manual trades as communities or cultures will perhaps seem strange. Yet communities of practitioners are connected by more than their ostensible tasks. They are bound by intricate, socially constructed webs of belief, which are essential to understanding what they do (Geertz, 1983). The activities of many communities are unfathomable, unless they are viewed from within the culture…. In a significant way, learning is, we believe, a process of enculturation. (Brown, Collins, & Duguid, 1989, p. 33)

COAT participants, as adult learners, found that the sometimes frustrating experience of being situated as a student within an unfamiliar, authentic online learning environment contributed to them rethinking their teaching practice.

The instructional approach of learning from the COAT facilitator modeling online teaching practice was informed by Bandura’s (1977) social learning theory. This approach aligns with cognitive apprenticeship (Brown, Collins, & Duguid, 1989). The idea of apprenticeship also appears in research focused on the teaching beliefs and practices of higher education instructors with Kane, Sandretto, and Heath (2002) arguing that

As preparation for teaching, university academics can be said to have completed an ‘apprenticeship of observation’ during their years as undergraduate and graduate students (Lortie, 1975). Their beliefs and conceptions of good teaching are a result of this apprenticeship and a ‘trial by fire’ in the lecture theatre, classroom, or laboratory. (p. 199)

Teaching online adds a new dimension to this trial by fire, and one focus group participant’s comments echoed the ideas in the above quotation:

I second the point about experiencing an online class as a student. We all sat in the traditional classroom, so we know what takes place - what we like and what we didn’t like. An instructor who has not taken an online class is at a real disadvantage.

Being situated as a student in an online course, that had content focused on online teaching theories and practice, allowed participants to observe the COAT facilitator model online teaching skills and strategies while she also participated in discussions about what she was doing and why she was doing certain actions.

In addition to learning from the COAT facilitator, alumni also learned from the other COAT participants, especially those who had prior online teaching experience. Over half of the focus group participants (13 out of 24) identified that taking COAT provided them with the opportunity to interact with other instructors and these interactions played a part in influencing later practice. The code Being part of a community of peers that emerged from the analysis of the data in this study is not a new concept and is embedded in the literature on communities of practice (CoP). Hildreth and Kimble (2008) argued that “Teaching is a very personal and ‘individual’ activity, yet teachers benefit greatly from links with other teachers, both with colleagues in their own establishment and with colleagues in the wider teaching community” (p. x).

Reflection on the learning that occurred in the community of COAT peers while they were immersed in an authentic online learning environment as students led to a review of the literature on situated cognition (Brown, Collins, & Duguid, 1989); CoP (Lave & Wenger, 1991); practice fields (Barab & Duffy, 2000, 2012); and collectives, networks, and groups (Dron & Anderson, 2007). Lave and Wenger (1991), in their book on situated learning, introduced CoP as nonacademic learning environments where novices learn from more experienced practitioners through legitimate peripheral participation, a form of apprenticeship. CoP as a concept has been applied to many fields since its inception, and Hildreth and Kimble (2008) considered it to have evolved into an “umbrella term” (p. xi) that now covers a range of group types with similar characteristics of being informal learning environments with voluntary membership of people interested in discussing practice and learning from each other while sharing resources and knowledge in a specific area. This knowledge is often tacit in nature.

Andriessen (2005), in his research into the classification of knowledge community archetypes, concluded that “the same term of ‘community of practice’ has been applied to different types of communities, that is, to strategic communities, to informal communities and to informal networks” (p. 209). According to Dron and Anderson (2007) “individuals join Networks to associate with others of like interest or vocation, or who know more, or who would like to learn similar things” (p. 2461). They differentiate between groups as formal, structured, closed phenomena and networks as being informal, unstructured, and open. The COAT course is an example of a group and the COAT Facebook presence that was created by a COAT participant, independent from the COAT project, is an example of an emerging network. The unexpected, off-topic, side discussions in the focus groups were a demonstration of COAT alumni’s desire to network informally.

In a similar manner, using Barab and Duffy’s (2012) definition of educational practice fields, “Contexts in which learners, as opposed to legitimate participants, can practice the kinds of activities that they will encounter outside of school” (p. 34), COAT can be seen to be a type of practice field where participants practice online teaching and learning activities and skills separate from their real-life teaching situations. However, what is missing from this picture of COAT as a practice field is the reality that COAT participants were also already legitimate participants in communities of educators, and some had extensive prior online teaching experience which was demonstrated in the learning afforded by the community of peers. Figure 2 draws together ideas of formal, semistructured, and informal learning spaces in relation to the concept that emerged from this study of immersion in an online learning environment helping participants become enculturated into the practice of online teaching and learning. In Figure 2, the left oval represents COAT participants being positioned as students in an authentic, structured, facilitator-led training course. Within that learning space, opportunities were designed for semistructured peer-to-peer discussions in the Cyber Café where colleagues, with varying levels of experience of online teaching, gathered around topics of their own choosing. Experiences with elements within the formal COAT course and the semistructured internal network of peers were identified as influencing later teaching practice. The area to the right of Figure 2 represents informal learning spaces that COAT alumni, as practitioners in the workplace and as members of communities of online educators, may choose to join to network and contribute to communities of practice.

The first principle for designing training for online instructors emerged from the concept of being immersed in an unfamiliar online learning environment:

Training for online instructors should be designed using a situated learning perspective that positions instructors as students in an authentic learning environment that is similar to the targeted teaching environment.

The recommendations that participants made for COAT to consider training for online teaching in nonLMS learning environments resonate with this design principle as the authentic learning environment could include any number of new and emerging learning technologies and social media. It is important to stress that all the design principles that emerged from this study are situational principles for designing faculty training. “Situational principles are ones that are not universal – they only apply in some situations. They exist on a continuum from situations that are very common (close to universal) to ones that are highly local (apply very rarely)” (Reigeluth & Carr-Chellman, 2009, p. 57). By including the voices of the diverse professionals who participated in COAT, it is hoped that practitioners working in similar situations to the instructors who participated in this study will find these design principles useful and possibly transferable to their teaching and learning environments.

Pools of Practice

All 24 focus group participants identified at least one element of COAT that influenced their subsequent online teaching practice. A primary influence was on course (re)design with 18 focus group participants making comments on how taking COAT had influenced their subsequent online course development, design, and redesign. Research participants also identified other influential elements of COAT such as learning about LMS features, pedagogy, online instructor role, and so on. In addition, other influences on online teaching practice were identified such as prior teaching experiences, other professional development opportunities, and institutional input. Using the metaphor of the COAT course being one of many pebbles making ripples in individual pools of practice helped to conceptualize some of the ideas captured in the data analysis. In the same way that a pebble is a concrete object with defined edges, COAT is a specific course with defined learning outcomes. The number and type of ripples from a pebble being thrown into a pool may be expected or unexpected depending on the situational circumstances. The impact of COAT on an individual’s practice may align with the defined learning outcomes of the training, but may be unexpected depending on the instructor and the teaching context. Figure 3 portrays that a training course such as COAT is one of many possible influences on the professional practice of individual instructors and that the ripples from a training course are diverse, specific to a particular teaching context, and may be unanticipated. Figure 3 shows some examples of other possible influences on practice, but more pebbles are possible depending on the instructor’s prior and current learning and work-related experiences.

(Re)designing online courses.

COAT was designed to prepare instructors to teach predesigned courses, and, although basic instructional design principles were introduced in the COAT course, the emphasis of the COAT content was on the delivery, not on the design of online courses. Some participants in this research study were disappointed with the lack of emphasis on designing courses. The data collected in this study highlighted that for many instructors course design issues were one of their major concerns, as they are asked to design and then teach online courses with limited, and sometimes nonexistent, prior online teaching and instructional design experience or institutional support. Second, instructors have an almost implicit responsibility to customize the course to meet the divergent and particular needs of students, thus many both want to and feel a responsibility to edit and improve existing courses that they are hired to teach. The assumption of the COAT project that participants would teach courses predesigned by teams proved to be incorrect. The findings demonstrated that COAT’s impact was broader than planned with an unexpected outcome being that a key takeaway from the COAT course was its impact on participants’ (re)designing online courses which highlighted the need for attention within the COAT project for offering optional skill development in course design. Participants in this study made suggestions that COAT could consider on how course design could be further explored either in the current course or in potential advanced courses.

Other professional practice.

Another unexpected outcome of participating in the COAT course was the impact the training had on professional practices other than online teaching practice. Four focus group participants commented that COAT had affected their campus-based teaching practice. In addition to COAT’s impact on campus-based teaching, six focus group participants said COAT had impacted their nonteaching practice in the areas of instructional design, managing online programs, and training faculty.

The second principle for designing training for online instructors emerged from the findings on unexpected outcomes from taking COAT:

Training for online instructors should prepare participants for diverse teaching situations which might include requirements to (re)design online courses and opportunities to teach in emerging learning environments.

Conclusion

The COAT project originated in the desire of a group of instructional designers, online faculty, and administrators from various institutions to collaboratively tackle the growing problem of how to best provide quality, accessible training for instructors who are making the transition to online teaching. Using a DBR methodological approach within an overall interpretivist research paradigm, this study evaluated whether the content, structure, and instructional approaches of the COAT course effectively helped instructors teach their subsequent online courses. Research participants identified that the experience of being situated as students in an authentic online course focused on online teaching and learning positively influenced their later online teaching, campus-based teaching, and nonteaching professional practice. This study provided detailed feedback for the COAT project, and the design principles that emerged from this study may be of interest to researchers and professionals who are involved in developing training for instructors who teach online. The findings from this study expand knowledge and contribute to the research literature on training for both experienced and inexperienced online instructors. Other recent studies focused on training for online instructors have resulted in similar findings to this study (see, for example, Eliason & Holmes, 2010; Kukulska-Hulme, 2012; MacDonald, 2010; McQuiggan, 2011; Storandt, Dossin, & Piacentini Lacher, 2012; Terantino & Agbehonou, 2012). By combining the findings from these research studies that were conducted in different contexts, the resulting design principles become more grounded in diverse situations and learning environments thus adding to the likelihood of transferability as effective design principles and practices to additional contexts.

References

Amiel, T., & Reeves, T. C. (2008). Design-based research and educational technology: Rethinking technology and the research agenda. Educational Technology and Society, 11(4), 29-40. Retrieved from http://www.ifets.info/journals/11_4/3.pdf

Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in education research? Educational Researcher, 41(1), 16-25. doi:10.3102/0013189X11428813

Andriessen, J. H. E. (2005). Archetypes of knowledge communities. In P. Van Der Besselaar, G. De Michelis, J. Preece, & C. Simone (Eds.), Communities and technologies 2005: Proceedings of the second communities and technologies conference, Milano 2005 (pp. 191-213). The Netherlands: Springer.

Bandura, A. (1977). Social learning theory. Englewood Cliffs, NJ: Prentice-Hall.

Barab, S. A., & Duffy, T. M. (2000). From practice fields to communities of practice. In D. H. Jonassen & S. M. Land (Eds.), Theoretical foundations of learning environments (pp. 25-55). Mahwah, NJ: Lawrence Erlbaum Associates.

Barab, S. A., & Duffy, T. M. (2012). From practice fields to communities of practice. In D. H. Jonassen & S. M. Land (Eds.), Theoretical foundations of learning environments (2nd ed., pp. 29-65). New York, NY: Routledge.

Barab, S. A., & Squire, K. (2004). Design-based research: Putting a stake in the ground. Journal of the Learning Sciences, 13(1), 1-14. doi.10.1207/s15327809jls1301_1

Bell, P. (2004). On the theoretical breadth of design-based research in education. Educational Psychologist, 39(4), 243-253. doi:10.1207/s15326985ep3904_6

Brookfield, S. (1993). Through the lens of learning: How the visceral experience of learning reframes teaching. In D. Boud, R. Cohen, & D. Walker. (Eds). Using experience for learning (pp. 21-32). Bristol, PA: The Society for Research into Higher Education & Open University Press.

Brown, A. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The Journal of the Learning Sciences, 2(2), 141-178. doi.10.1207/s15327809jls0202_2

Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32–42. Retrieved from http://edr.sagepub.com/content/18/1/32

Cohen, L., Manion, L., & Morrison, K. (2007). Research methods in education (6th ed.). New York, NY: Routledge.

Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoretical and methodological issues. Journal of the Learning Sciences, 13(1), 15-42. doi.10.1207/s15327809jls1301_2

Crotty, M. (1998). The foundations of social research: Meaning and perspective in the research process. Thousand Oaks, CA: Sage.

Dede, C., Ketelhut, D. J., Whitehouse, P., Breit, L., & McCloskey, E. M. (2009). A research agenda for online teacher professional development. Journal of Teacher Education, 60(1), 8-19. doi:10.1177/0022487108327554

Denzin, N. K., & Lincoln, Y. S. (2011). Introduction: The discipline and practice of qualitative research. In N. K. Denzin & Y. S. Lincoln (Eds.), The Sage handbook of qualitative research (4th ed., pp. 1-19). Thousand Oaks, CA: Sage.

Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5-8. doi.10.3102/0013189X032001005

Dron, J., & Anderson, T. (2007). Collectives, networks and groups in social software for e-learning. In T. Bastiaens & S. Carliner (Eds.), Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2007 (pp. 2460-2467). Chesapeake, VA: AACE.

Dubins, B. H., & Graham, M. B. (2009, August). Training instructors to teach online: Research on competencies/best practices. Paper presented at the 25th Annual Conference on Distance Teaching and Learning, Madison, WI. Retrieved from http://www.uwex.edu/disted/conference/Resource_library/proceedings/09_20433.pdf

Eliason, S. K., & Holmes, C. L. (2010). Reflective practice and inquiry in professional development for online teaching. MERLOT Journal of Online Teaching and Learning, 6(2), 454-465. Retrieved from http://jolt.merlot.org/vol6no2/eliason_0610.pdf

Friese, S. (2012). Qualitative data analysis with Atlas.ti. Thousand Oaks, CA: Sage.

Guskey, T. R. (2000). Evaluating professional development. Thousand Oaks, CA: Corwin Press.

Hildreth, P., & Kimble, C. (2008). Introduction and overview. In C. Kimble, P. Hildreth, & I. Bourdon (Eds.), Communities of practice: Creating learning environments for educators, Volume 1 (pp. ix-xix). Charlotte, NC: Information Age Publishing.

Kane, R., Sandretto, S., & Heath, C. (2002). Telling half the story: A critical review of research on the teaching beliefs and practices of university academics. Review of Educational Research, 72(2), 177-228. Retrieved from http://rer.sagepub.com/content/72/2/177.short

Ketelhut, D. J., McCloskey, E. M., Dede, C., Breit, L, A., & Whitehouse, P. L. (2006). Core tensions in the evolution of online teacher professional development. In C. Dede (Ed.), Online professional development for teachers: Emerging models and methods (pp. 237-263). Cambridge, MA: Harvard Education Press.

Koro-Ljungberg, M., Yendol-Hoppey, D., Smith, J. J., & Hayes, S. B. (2009). (E)pistemological awareness, instantiation of methods, and uninformed methodological ambiguity in qualitative research projects. Educational Researcher, 38(9), 687-699. doi:10.3102/0013189X09351980

Krueger, R. A. (1988). Focus groups: A practical guide for applied research. London, United Kingdom: Sage.

Krueger, R. A., & Casey, M. A. (2009). Focus groups: A practical guide for applied research (4th ed.). Thousand Oaks, CA: Sage.

Kukulska-Hulme, A. (2012). How should the higher education workforce adapt to advancements in technology for teaching and learning? The Internet and Higher Education, 15(4), 247-254. doi:10.1016/j.iheduc.2011.12.002

Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. New York, NY: Cambridge University Press.

MacDonald, J. (2010). Activity design in online professional development for university staff. European Journal of Open, Distance and E-Learning, 1. Retrieved from http://www.eurodl.org/materials/contrib/2010/Janet_Macdonald.pdf

McKenney, S. E., & Reeves, T. C. (2012). Conducting educational design research. New York, NY: Routledge.

McQuiggan, C. A. (2011). Preparing to teach online as transformative faculty development (Doctoral dissertation). Retrieved from ProQuest Dissertations & Theses: Full Text. (ProQuest Document ID: 887709387).

Moss, P. A., Phillips, D. C., Erickson, F. D., Floden, R. E., Lather, P. A., & Schneider, B. L. (2009). Learning from our differences: A dialogue across perspectives on quality in education research. Educational Researcher, 38(7), 501-517. doi:10.3102/0013189X09348351

Nicholas, D. B., Lach, L., King, G., Scott, M., Boydell, K., Sawatzky, B. J., … Young, N. L. (2010). Contrasting internet and face-to-face groups for children with chronic health conditions: Outcomes and participant experiences. International Journal of Qualitative Methods, 9(1), 105-121. Retrieved from http://ejournals.library.ualberta.ca/index.php/IJQM/article/view/4080/6497

Reigeluth, C. M., & Carr-Chellman, A. A. (2009). Situational principles of instruction. In C. M. Reigeluth & A. A. Carr-Chellman (Eds.), Instructional-design theories and models (Vol. III) (pp. 57-68). New York, NY: Routledge.

Saldaña, J. (2009). The coding manual for qualitative researchers. Thousand Oaks, CA: Sage Publications.

Sandoval, W. A., & Bell, P. (2004). Design-based research methods for studying learning in context: Introduction. Educational Psychologist, 39(4), 199-201. doi:10.1207/s15326985ep3904_1

Shattuck, J. (2013). Training higher education adjunct faculty to teach online: A design-based research study (Unpublished doctoral dissertation). Centre for Distance Education, Athabasca University, Edmonton, AB. Retrieved from https://dt.athabascau.ca/jspui/handle/10791/27

Shattuck, J., Dubins, B., & Zilberman, D. (2011). MarylandOnline’s inter-institutional project to train higher education adjunct faculty to teach online. International Review of Research in Open and Distance Learning, 12(2), 40-61. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/933

Storandt, B. C., Dossin, L. C., & Piacentini Lacher, A. (2012). Toward an understanding of what works in professional development for online instructors: The case of PBS teacherline. Journal of Asynchronous Learning Networks, 16(2), 121-162. Retrieved from http://www.pbs.org/teacherline/courses/common_documents/research/toward-an-understanding-of-pd.pdf

Strauss, A., & Corbin, J. (1998). Basics of qualitative research: Techniques and procedures for developing grounded theory. Thousand Oaks, CA: Sage.

Terantino, J. M., & Agbehonou, E. (2012). Comparing faculty perceptions of an online development course: Addressing faculty needs for online teaching. Online Journal of Distance Learning Administration, 14(2). Retrieved from http://www.westga.edu/~distance/ojdla/summer152/terantino_agbehonou152.html

Turney, L., & Pocknee, C. (2005). Virtual focus groups: New frontiers in research. International Journal of Qualitative Methods, 4(2), 32-43. Retrieved from http://ejournals.library.ualberta.ca/index.php/IJQM/article/view/4445

Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research and Development, 53(4), 5-23. doi.10.1007/BF02504682