The Impact of 'Virtualization' on Independent Study Course Completion Rates: The British Columbia Open University Experiment
VOL. 23, No. 1, 49-70
Abstract
In 1997 the British Columbia Open University (BCOU) adopted a virtualization strategy based primarily on twinning off-line independent study distance education courses (textbook-based with study guide and telephone and e-mail tutor support) with alternative online versions (textbook-based with integrated conferencing and communications provided through a learning management system). The intent was two-fold: (a) To increase course completion rates to British Columbia (BC) system levels, and (b) To establish a strong online presence, rapidly and cost effectively. ‘Course-twinning’ also provided an opportunity to assess the impact of online delivery on independent study course completion rates. In this study we benchmarked successful completion rates (SCRBs) for BCOU’s online academic courses and compared those to off-line course benchmarks.
Online SCRBss are consistently higher than off-line SCRBs. Online SCRBs gains follow a clear pattern: they increase with course level and subject matter classifications and are particularly large at the upper-level and in more quantitative disciplines (up to 21.2% increase). Online successful completion rates are comparable to BC system performance statistics in the credit course areas but fall well short for Adult Basic Education (ABE) courses. Since evidence from a contemporary course survey indicates that BCOU student participation in online course versions is not linked to demographic characteristics associated with successful completion rates, we suggest that BCOU student success online results primarily from the benefits of online course delivery. We discuss the implications of these results.
Résumé
En 1997, l’Open University de Colombie-Britannique (BCOU) adoptait une stratégie de virtualisation basée principalement sur l’action de jumelage des cours hors-ligne à distance (basés sur un livre ave un guide d’étude et du soutien tutoral par téléphone ou courriel) avec des versions alternatives en ligne (basés sur un livre avec de la visioconférence et des outils de communication intégrés fournis grâce à un système de gestion de l’apprentissage). Cette stratégie était à la base de deux intentions : a) Augmenter les taux de complétion des cours pour atteindre ceux du système de la Colombie-Britannique, et b) Établir une forte présence en ligne rapidement et à peu de frais. Le jumelage des cours permettait aussi d’évaluer l’impact de la diffusion en ligne sur les taux de complétion des cours hors-ligne. Dans cette étude, nous avons étalonné les taux de complétion réussie (SCRBs) pour les cours académiques en ligne de BCOU et les avons comparés à l’étalonnage des cours hors-ligne.
Les SCRBs des cours en ligne sont systématiquement plus haut que ceux des cours hors-ligne. Les gains des SCRBs des cours en ligne suivent un pattern clair : ils augmentent avec les classifications du niveau et du sujet du cours et sont particulièrement élevés aux niveaux supérieurs et dans les disciplines plus quantitatives (jusqu’à 21,2 % d’augmentation). Les taux de complétion réussie en ligne sont comparables à ceux des statistiques de performance du système de la Colombie-Britannique pour les cours crédités, mais sont de beaucoup inférieurs pour les cours de base en éducation des adultes. Étant donné que les résultats d’un sondage récent indiquent que la participation des étudiants aux cours en ligne n’est pas liée à des caractéristiques démographiques, nous suggérons que le succès des étudiants de BCOU en ligne résulte principalement des bénéfices de la diffusion en ligne des cours. Nous discutons les implications de ces résultats.
Introduction
The British Columbia Open University (BCOU) Course Completion Challenge
Historically, distance education course completion rates have been low compared to ‘traditional’ face-to-face courses (Beatty-Guenter 2001, Carr 2000, Holmberg 1995, Howell et al., 2004, Keegan 1986, Munro 1991, Paul 1986, 1990, Zajkowski 1997). When in 1997 the Open Learning Agency (OLA), BCOU’s parent institution, published preliminary research findings suggesting that BCOU’s online independent study course completion rates were 10 percentage points higher than their off-line counterparts (Eliesen and Black 1997, 1998a, 1998b), online course delivery became central to BCOU’s goal of increasing course completion rates “by an amount sufficient to ensure that within two years (i.e., by March 1999) they meet or exceed the minimum as established by the other institutions in the British Columbia College, Institute and Agency system (controlling for program area and using system-wide key performance indicators.)” OLA (1997, p. 3, as outlined in OLA’s Ends Policies. 1997 to 2000.)
To achieve this Ends Policy, the BCOU adopted a virtualization strategy based on two components: (a) Supplementing its off-line courses with e-mail communications, and (b) Converting off-line independent study courses (textbook-based with instructional resources, and telephone and e-mail tutor support) with twin online course versions (textbook-based with instructional resources and integrated conferencing and communications provided through a learning management system), intending for students to choose between an online and an off-line course version for as many courses as possible. “Course-twinning” ensured that students who did not have access to, or did not wish to use online technology, retained access to an equivalent off-line (print-based) course version in terms of textbooks, assignments, exams, marking guides, tutors, credit transfer, etc. (See more details below.) This approach was expected to increase course completion while protecting enrolment since the flexible course delivery features valued by open learning students were retained: open admission, continuous enrolment, flexible start and end dates, generous completion and extension periods, etc. (Black et al., 1994; Black, 1995; Black, 1996; Butler & Bischoff, 1999; Giguère, 2002). To achieve this outcome in a cost-effective manner, the institution developed software solutions such as “Oscar” (Klassen et al., 1999; Porter, 2001) and “Trucourse” (Friesen et al., 2007) that ‘automatically’outputted course assignments and course units to both off-line and online formats. Furthermore, by re-purposing existing off-line instructional resources through Oscar and Trucourse, the institution endeavoured to establish a strong online presence rapidly and cost effectively.
The BCOU as a Natural Experiment
Because of its course-twinning virtualization strategy and because BCOU assets were frozen between 2002 and 2005 in preparation for BCOU’s acquisition by another institution, the BCOU represents a natural experiment to assess the impact of online delivery on independent study course completion rates. BCOU academic policies and procedures, curriculum, staff and faculty, budget, and business development activities were frozen between February 2002 and April 2005, and the university came as close as is possible to the scientific experimental motto of ‘holding everything else constant’. Since this period of stability occurred while the BCOU was about half way through implementing its course-twinning strategy, many of its courses existed in ‘equivalent’ twin online and off-line versions. By benchmarking BCOU online and off-line independent study course completion rates during this period using a standard approach (author, 2007), this article assesses the impact of online delivery on independent study course completion rates and checks on BCOU’s progress toward its goal of increasing course completion rates to BC system levels.
The British Columbia Open University
In Canada, education is a provincial or territorial jurisdiction. At the beginning of this millennium, the Province of British Columbia’s Ministry of Advanced Education funded and supported 25 postsecondary institutions of which the BCOU was a constituent member. At that time, BCOU’s course enrolment ranged between 20-25,000 and it served 12-14,000 students annually before it ceased to operate on March 31, 2005; its operations were then taken over by Thompson Rivers University (TRU).
In its final days, the BCOU was a comparable institution to TéléUniversité (Québec) and Athabasca University (Alberta). It was a single mode distance education institution offering flexible learning opportunities through credit banking and distance education. BCOU’s course enrolment was slightly less than TéléUniversité and significantly less than Athabasca University (by a factor of about two and a half).
The British Columbia Open University Independent Study Courses
As an open learning institution, the BCOU offers independent study courses to its student population. The courses rely extensively on textbooks, print-based assignments, course units, as well as audio and video supplements mailed to students in a course package.
The main difference between the off-line and online course versions was in the the type of student-instructor and student-student communication available. For off-line course versions, students were able to contact tutors or staff via mail, telephone, or e-mail. Internet access was not required, and students were not provided with an opportunity to contact other students. Telephone calls were accepted by tutors during scheduled twice-weekly office hours and voice mail was provided in between office hours. Tutors were required to respond to student queries within two working days, as well as to return submitted assignments within three working days. Though some students participated in BCOU off-line independent study courses without using e-mail, e-mail communication has become commonplace and tutor telephone contacts have decreased by a factor of at least ten since students started to use the Internet to contact tutors. For online course versions, Internet access was required. The First Class or WebCT learning management system provided access to library and administrative support, to the tutor, to student discussions and conferences moderated by the tutor, and to ‘students only’ conferences. Online courses still relied extensively on textbooks and instructional materials mailed to students in a course package but less extensive materials (such as assignments and general instructions) were provided online. Turnaround time standards for conference messages and assignments were the same as for off-line courses. Thus the primary difference between off-line and online course versions was that the latter (a) Supports group communications facilitated by a tutor and (b) Provides students with an online learning environment that integrates academic and administrative communication and where other students, staff, and tutors are a couple of ‘clicks’ away. A few BCOU online independent study courses were also provided in a paced format. BCOU’s course pacing is not tutor led however, but rather the result of superimposing deadlines on an independent study course structure: 17 courses out of 72 courses existed in paced format as well as independent study format at some point during the 3 years covered by this study.
Methods
Data
The target population was students in academic areas (Arts, Science, and Business) who completed an independent study distance education course in the fiscal years 2002/03, 2003/04, and 2004/05 (this excludes laboratory courses.) During that period, the BCOU offered 70 online courses and 137 off-line courses, a balance that should enable us to compare online courses to off-line courses. The analysis comprises 203 data points based on 70 courses and 6,634 course enrolments. A datum represents a unique course delivery method and individual tutor combination including six or more students in a given year.
Experimental Design
Course completion rates were calculated on an annual basis for each unique course delivery method and individual tutor combinations. Stepwise linear regression was used with 16 institutional factors or sub-factors (independent variables) to identify which were most strongly associated with course completion (dependent variable). Course completion rate benchmarks were then established based on regression factors that were found to be statistically significant, rather than choosing them subjectively. We used a 0.05 level of statistical significance. This methodological phase of the research project is described in author (2007).
There was an opportunity to compare online benchmarks either to their off-line equivalent (a paired design targeting the same 70 courses) or to off-line courses overall (137 courses of which only 70 have a directly equivalent online version). We opted for the latter approach because it provided greater statistical power for two reasons: 1) It nearly doubled the number of courses (from 70 to 137) used to benchmark based on institutional factors (author, 2007), and 2) It included all 70 online courses as a basis for comparison. Using all 70 online courses would not be possible in a paired design because the basic datum consists of a course supported by a particular tutor in a particular course version in a given year. Consequently, the sub-set of courses where off-line and online course versions are supported by the same tutor is considerably less than 70. Thus, we felt that maximizing sample size for benchmarking purposes was the best approach to compare online courses to off-line courses.
Defining Completion Rates
One of the challenges in assessing the impact of virtualization on course completion rates is that results are difficult to compare because online effects are confounded with other intra- and inter-institutional differences, including how to measure completion rates (see Howell et al., 2004). Though ‘traditional’ institutions and ‘open and distance education’ institutions use distinctive grading policies particularly with regard to the definition of an incomplete or a withdrawal grade, the two systems can be linked through the concept of stable enrolment, viz., students who demonstrably started a course. Whereas most BC institutions identify ‘starters’ using a time-dependent approach (a cut-off date chosen so as to ensure that statistics are based on students who clearly made an attempt to start a course), Open Universities such as the BCOU identify starters with respect to outcome, viz., the fact that a student has completed some graded academic work, regardless of date. Through the concept of stable enrolment, the successful completion rate of “starters” links the grading systems of ‘open and distance education’ institutions and of ‘traditional’ institutions and their comparative performance (Woodley and Parlett 1983, Paul 1990, Howell et. al, 2004).
BCOU’s successful completion rates (SCR) were calculated as follows:
The total number of students who obtained a passing grade on an ABCD scale, divided by the total number of students who started (i.e., the number of students who obtained a passing grade on the ABCD scale plus those who received an Incomplete or a Fail grade) times 100.
This course completion rate definition excludes “W” grades as defined by ‘open and distance education’ institutions like the BCOU, i.e., a grade given to students who do not submit any academic work for grading. The BCOU “W” grade is not included in GPA calculations and as such bears no academic penalty.
Coding Institutional Factors
The institutional factors consist of course factors and tutor factors. The course factors are: Course Level, Subject Matter, Number of Credits, Year of Delivery, Delivery Pacing, as well as a unique number for each course (Course Number). Course Level is coded as follows:
| Course Level | Regression Coding |
| Subfactor 1: | Non-credit (ABE*) = 0 and Credit = 1 |
| Subfactor 2: | ABE = 0, and 1 to 4 for university from 100-level to 400-level (year 1 to year 4) |
| Subfactor 3: | Introductory course =1 Other courses = 0 (Introductory is first of a pair of first- or second-year courses, such as Introductory Physics 1 and Introductory Physics 2) |
ABE = Adult Basic Education
Subject Matter is coded as follows: Applied Academic areas (Business Administration and Business computer courses) are coded as 1. Liberal Arts subjects are coded from 2 to 4 according to subjective levels of quantitative or scientific contents.
| Subject Matter | Disciplines | Regression Coding |
| Business | Administration, Business, Workplace Training | 1 |
| Humanities | English, French, History, Humanities, Spanish | 2 |
| Social Sciences | Economics, Geography, Geology, Political, Science, Psychology, Sociology | 3 |
| Science and Mathematics | Biology, Environmental Studies, Environmental Science, Chemistry, Mathematics, Physics | 4 |
Delivery Pacing is coded as 1 (Independent study without pacing) and 2 (Independent study with pacing, i.e., with assignment due dates).
The tutor factors are: Course Load (the number of courses supported by a tutor for that year), Student Load (the number of students supported by a tutor for that year), Age (in 2002), Years of Service at OLA, Gender, Place of Residence, Academic Qualifications, as well as a unique number for each tutor (Tutor Number). Place of Residence and Academic Qualifications are coded as follows:
| Tutor Factors | Regression Coding |
| Place of Residence in 2002/03 | 1 = Metropolitan: Greater Vancouver and Greater Victoria 2 = Other urban centres: Kelowna, Kamloops, Nanaimo, Prince George 3 = Rural: All other locations |
| Academic Qualifications | 1 = Phd, 2 = Master's, 3 = Bachelor's or below |
For more information see Giguère (2007).
Results and Interpretation
Identifying Benchmarking Factors from Among Institutional Factors
We regressed 16 institutional factors on 3 years of online independent study course completion rate data. Six regression steps identified 6 factors accounting for 42.3% of SCR variation (Table 1). This is a relatively high percentage considering that completion rates ranged widely (minimum 20.0% and maximum 100.0%). In the corresponding off-line course benchmark study (author, 2007), eight regression steps accounted for 46.8% of SCR variation.
Table 1: Statistical Regression of Institutional Factors on Online Course SCRs (Stepwise Linear Regression Method)
Regression Step |
Factors |
F to Enter |
R2 proportion of variation accounted for by regression |
1 |
Course Level: Subfactor 2 ABE to 400-level | 69.6 |
0.259 |
2 |
Course Level: Subfactor 1 Credit vs Non-credit | 41.5 |
0.295 |
3 |
Course Level: Subfactor 3 Introductory or Not | 37.6 |
0.364 |
4 |
Tutor Gender | 31.4 |
0.391 |
5 |
Course Number | 27.0 |
0.409 |
6 |
Year of Delivery | 23.7 |
0.423 |
Overall this regression is comparable to the one for BCOU off-line courses (Giguère, 2007) except that Subject Matter and Tutor Course Load did not enter into consideration for the online course regression. One factor that did not contribute to the online course regression model presented here in spite of high expectation was Delivery Pacing (coefficient = 0.001, t = .009, p > 0.50).
The first two Course Level factors entering at step 1 and 2 of the regression was used to benchmark online course completion rates for two reasons. First, they account for 29.5% of the statistical variation before tutor specific or course specific factors enter the regression. Second, they match two Course Level factors used to benchmark off-line course SCRs (Giguère, 2007).
Step 2 of the linear regression can then be used to determine online course SCRBs according to the general regression formula:
Y = a + (b . X1) + (c . X2) |
Equation 1 |
where X1 represents the slope for Course Level: Subfactor 2 codes (ABE to 400-level) and X2 the slope for Course Level: Subfactor 1 codes (Credit vs Non-credit).
Equation 2 generates the following online course SCRBs, shown in Table 2 with actuals:
SCRB = 53.1+ (7.0 . X1) + (14.6 . X2) |
Equation 2 |
Table 2: BCOU Online Successful Course Completion Rates Benchmarks*, by Course Level: SCRBs and SCR Actuals (sample size in parenthesis**)
Non-Credit Courses |
Credit Courses |
||||
ABE |
100-Level |
200-Level |
300-Level |
400-Level |
|
| SCRBs (from Equation 1) | 53.1 |
74.7 |
81.7 |
88.7 |
95.7 |
| SCR Actuals | 53.1 (13) |
75.8 (107) |
76.5 (40) |
93.3 (27) |
94.9 (16) |
* SCRBs, expressed as a %, for academic courses offerings during 2002/03 through 2004/05 period
** A datum is based on an SCR percentage for a unique course version and tutor combination for a given year
The fit between the SCRBs (from the linear regression model) and SCR actuals is within a few percentage points in each case. Two differences are near 5 percentage points (200- and 300-level courses), one on the high side and one on the low side. Actuals suggest three groupings: (a) ABE, (b) 100- and 200-level, and (c) 300- and 400-level courses. The benchmarks on the other hand show a gradual progression across all levels, since they are based on a linear regression model. Nevertheless, the regression model organizes online SCRBs congruently and effectively across five Course Levels, as was the case for off-line SCRBs (Giguère, 2007).
Other Significant Regression Factors
Factors entering the regression in steps 3 through 6 account for variation over and above the variation accounted for by the proposed benchmarking factors, Course Level and Subject Matter classifications. They are Introductory Course (coefficient = -9.8, t = -4.23, p < 0.001; i.e., online SCRs are on average 9.8% lower for introductory courses), Tutor Gender (coefficient = -6.3 , t = -3.17, p = 0.002; i.e., online SCRs are on average 6.3% lower for male tutors than for female tutors), Course Number (coefficient = 0.052, t = 2.66, p = 0.008), and Year of Delivery (coefficient = -2.6, t = -2.18, p < 0.031; i.e., online SCRs decrease on average 2.6% per year). The Introductory Course and Gender effects are nearly double those observed in the off-line study (decreases of 5.5% and 3.2% for Introductory Course and Gender respectively, Giguère, 2007). More surprisingly however, Year of Delivery has a negative effect for online course SCRs, whereas it has a positive effect for off-line course SCRs (1.9, t = 2.8, p < 0.01) (Giguère, 2007). This suggests that off-line SCRs increased while online SCRs decreased during the 2002-2005 study period. The averages are shown in Table 3.
Table 3: BCOU Online and Off-line SCRs (average) Over Three-Year Experimental Period (sample size in parenthesis*)
Year |
2002/03 |
2003/04 |
2004/05 |
| Online SCRs | 81.4 (53) |
78.5 (68) |
76.2 (82) |
| Offline SCRs** | 66.5 (200) |
70.2 (186) |
69.8 (162) |
| SCR Difference (Online minus Offline) | 14.9 |
8.3 |
6.4 |
Thus two components contributed to the BCOU’s virtualization strategy from 2002/03 through 2004/05, a small SCR gain due an off-line SCR increase over time (author, 2007) and a large SCR gain due to high online course SCRs relative to off-line course SCRs. Both components contribute toward BCOU’s 1997 goal of increasing course completion rates to BC system levels.
To investigate declining online course SCRs from years 2002/03 through 2004/05, we tested for interaction between Year of Delivery and all other significant regression factors. Only one interaction was statistically significant, i.e., a Year of Delivery by Gender interaction (t = -3.97, p < 0.0001). Table 4 displays average SCRs by Gender and Year of Delivery.
Table 4: BCOU Online SCRs (average) by Gender, Over Three-Year Experimental Period (sample size in parenthesis*)
Year |
2002/03 |
2003/04 |
2004/05 |
| Female | 83.5 (23) |
79.6 (22) |
80.8 (28) |
| Male | 79.9 (30) |
78.1 (46) |
73.8 (54) |
| Difference (Female minus Male) | 3.6 |
1.5 |
7.0 |
* A datum is based on an SCR percentage for a unique course version and tutor combination for a given year
While female tutors maintained high online course SCRs over the 2002-2005 experimental period, male tutor’s SCR declined by 6 percentage points. The gender gap grew from 3.6 percentage points in 2002-2003 to 7.0 percentage points in 2004-2005.
Since the tutor gender distribution is very similar among online and off-line courses over the experimental period (Table 5; this would be expected in an institution where assets are frozen and the tutor population stable), the observed gender SCR difference cannot be associated with a gender distribution change among online and off-line course versions during that period.
Table 5: Gender Distribution Among Online and Off-line Course Versions Between 2002-03 and 2004-05, Wilcoxon Signed Ranks Statistic (Z)
Year |
Male |
Female |
| Online | Z = -1.177 p = 0.239 |
Z = -0.051 p = 0.959 |
| Online | Z = -0.155 p = 0.877 |
Z = -0.311 p = 0.756 |
We conclude that male tutors are associated with declining online SCRs over the experimental period, though the reasons for this are not clear. Anecdotal evidence gathered in the context of a performance evaluation of Arts and Science tutors suggests that male tutors favor the instructional approach used in off-line course delivery, an approach that consists in marking and providing feedback in response to student requests, rather than proactive facilitation. A transfer of these off-line tutoring skills to online environments by male tutors may have resulted in lower participation levels in online conferences and thus may have resulted in reducing the effectiveness of BCOU's online delivery by male tutors.
Comparing Online and Off-line SCRBs
Table 6 expresses results as online SCR gains, i.e., as the difference between online and off-line SCRBs. This is obtained by subtracting SCRBs in Table 2 above, from the author’s (2007) Table 3.
Table 6: SCRB Gains Online (Percentage Point Difference Between Online and Off-line SCRBs), By Course Level and Subject Matter
Subject Matter |
Non-Credit Courses |
Credit Courses |
|||
ABE |
100-Level |
200-Level |
300-Level |
400-Level |
|
| Business | 0.0 |
2.1 |
4.3 |
6.5 |
|
| Humanities | -1.4 |
3.7 |
5.9 |
8.1 |
10.3 |
| Social Sciences | 2.4 |
7.4 |
9.6 |
11.8 |
14.0 |
| Science and Math | 6.1 |
11.1 |
13.3 |
15.5 |
|
Table 6 yields three key findings.
- SCRB gains online show a strong and coherent pattern across Subject Matter and Course Level classifications;
- SCRB gains online are positive for all cells but two, and are higher the higher the Course Level and the more academic and quantitative the Subject Matter; and
- SCRB gains online are as large as 15.5 percentage points. This represents an increase of up to 21.2% over corresponding off-line SCRBs.
However, since BCOU students can select online or off-line course versions at the time of course registration, it is possible that Table 6’s differences result in whole or in part from the fact that students with stronger academic abilities preferentially select online delivery. For example students with more prior education could be more successful than students with less prior education and choose online course delivery more frequently than students with less prior education; if this were the case, student behaviour would produce an ‘apparent’ online SCR increase (by apparent we mean SCR increases due to differential student distribution among course versions rather than to the benefits of online course delivery features). If ‘apparent’ online SCR gains were to occur, we should observe (a) demographic differences among BCOU online and off-line student population, (b) a correspondence between online SCR gains and significantly different demographics, as per Table 6’s results, and (c) an over-representation of demographics associated with student success in the online student population. Results from a contemporary course survey allow us to test this hypothesis. The OLA Course Survey includes information about six student demographic characteristics as well as course completion rate data. The survey is based on 8,633 respondents who finished an online or off-line academic course between September 1, 2004 and December 31, 2006.
1) Do demographic differences exist among BCOU online and off-line student population?
Using a t-statistic for parametric data, Mann-Whitney’s z-statistic for ordinal data, and χ2 statistic for nominal data, we find (Table 7) that two out of six student demographics are statistically different among the online and off-line student populations (Prior Education, Employment Status). Two others yielded probability values between 0.05 and ≤ 0.10 (Gender and “In BCOU Program”) and should be considered for further investigation.
Table 7: Comparative Demographics for BCOU Online and Off-line Student Populations (September 2004-December 2006 Course Survey)
Student Demographic Characteristics |
Offline |
Online |
Difference |
Significance |
| Age (years) | 32.1 |
32.8 |
0.7 |
T = -1.19 P = 0.234 |
| Gender (% male) | 28.8 |
32.4 |
3.6 |
χ2 = 6.995 p = 0.072 |
| In BCOU Program (%) | 28.9 |
46.6 |
17.7 |
χ2 = 6.504 p = 0.090 |
| Prior Education* | 10.1 |
10.3 |
0.2 |
Z = -2.149 P = 0.032 |
| Employment Status** | 1.83 |
1.93 |
0.1 |
Z = -3.19 P = 0.001 |
| Region (% metropolitan) | 53.2 |
55.0 |
1.8 |
χ2 = 5.046 p = 0.168 |
*Coding based on number of years of education (elementary = 7, grade 12 = 12, 2-year
Diploma = 14, 4-year degree = 16, etc.) Undeclared = 0
**Coding: Undeclared = 0, unemployed = 1, student = 2, employed = 3
Continuing with the two statistically significant student demographics, we consider the second expected observation:
2) Is there correspondence between SCR gains online and statistically significant student demographic differences, according to a pattern that is consistent with Table 6’s results?
Table 8 displays online SCR gains according to significant demographic differences, by Course Level (top) and Subject Matter (bottom). Only “Prior education” yields a pattern that is consistent with Table 6 and a positive answer to our question.
Table 8: SCR Gains Online Across Course Level and Subject Matter Classifications for Statistically Significant Demographic Characteristics (Surveyed Population, September 2004-September 2006)
Course Level |
|||||
ABE** |
100-Level |
200-Level |
300-Level |
400-Level |
|
| Prior Education+ | -- |
1.9 |
-8.0 |
6.1 |
8.8 |
| Employment Status++ | -- |
-4.2 |
-7.8 |
3.2 |
-6.0 |
Subject Matter |
|||||
Business |
Humanities |
Social Sciences |
Science and Math** |
||
| Prior Education+ | 4.2 |
2.0 |
-3.5 |
-- |
|
| Employment Status++ | 0.8 |
3.1 |
-4.3 |
-- |
|
**Insufficient data for online courses
+University course work versus no university course work
++Employed versus unemployed or student
Continuing with “Prior Education”, we consider the third expected observation:
3) Are there proportionally more university-educated students in online course versions, and particularly so as Course Level increases?
We used the Kolmogorov-Smirnov one-tailed test for large samples (Siegel, 1956, p. 131) to test whether students with prior university education are over-represented in online course versions. We observed neither a statistically significant difference (Chi-square value of 3.14, 0.30 > p > 0.20), nor a proportional increase along a Course Level classification (Table 9).
Table 9: Proportional Distribution of Students with Prior University Education* Among Online and Off-line Student Populations, Across Course Levels (Surveyed Population, September 2004-September 2006)
Course Level |
|||||
University Level Course Work |
100-Level |
200-Level |
300-Level |
400-Level |
Sample Size |
| A-Online (% of total) | 0.342 |
0.349 |
0.260 |
0.333 |
1,082 |
| B- Offline (% of total) | 0.381 |
0.309 |
0.300 |
0.301 |
6,840 |
| Difference (A-B) | -0.039 |
0.040 |
-0.040 |
0.032 |
|
*Demographic characteristic statistically different among online and off-line student populations and exhibiting online SCR gains consistent with Table 6.
In short, students with prior university education did not choose online course delivery in a greater proportion than students without prior university education. In the end, we were not able to find evidence that BCOU student's participation in online course versions is linked to demographic characteristics or academic abilities associated with SCRs. This increases the likelihood that the gains online exhibited in Table 6 reflect the greater effectiveness of the BCOU's independent study online delivery model. We suggest, therefore, that in most BCOU independent study courses, the features of online course delivery helped students achieve high SCRs. We also suggest that the effectiveness of independent study online delivery is greatest with student populations engaged in upper levels of instruction and in more quantitative subject matters.
Table 10. SCRs for the BC College, Institute and Agency System and BCOU Off-line and Online Academic Courses for 2002-2005 period (sample size in parenthesis*)
ABE |
Lower Level |
Upper Level |
|
| BC College, Institute and Agency system, 1998 | 72
|
79
|
93 |
| BCOU online SCRs, 2002/05* | 53.1
(13) |
76.0
(147) |
93.9 (43) |
| BCOU offline SCRs, 2002/05* | 49.2 (103) |
70.0 (336) |
83.3 (109) |
*A datum is based on an SCR percentage for a unique course version and tutor combination for a given year
Though BCOU off-line course SCRs fall well short of BC system performance at all levels, BCOU’s online courses SCRs fare well in the credit course area. BCOU’s course twinning strategy had a negligible effect on SCRs for non-credit courses, i.e., in the ABE area. BCOU online SCRs exceed BC system statistics for upper-level credit courses and fall slightly short for lower-level credit courses. These results demonstrate near equivalency between BCOU’s independent study online credit courses and face-to-face courses offered by BC public institutions, when using SCR statistics based on a concept of stable enrolment. Our results are in agreement with other studies that found high completion rates for online course delivery, including claims of ‘equivalency’ with face-to-face course delivery (Card & Horton 1998, Fredda 2000a, 2000b, 2000c, Johnson et al. 2000, Machtmes & Asher 2000, McDonald & Gibson 1998, Rosberg 1997, Russell 1999, Sener 1996). Our study specifically demonstrates that online independent study course designs can also achieve near equivalency with comparable face-to-face courses.
Assessing BCOU’s Virtualization Strategy
Our results demonstrate that the BCOU virtualization strategy achieved part of the goal of increasing course completion rates “by an amount sufficient to ensure that within two years (i.e., by March 1999) they meet or exceed the minimum as established by the other institutions in the British Columbia College, Institute and Agency system (controlling for program area and using system-wide key performance indicators.)” OLA (1997). Though this goal was not achieved on schedule (i.e., by March 1999), our study suggests that online credit courses achieved this performance level as they were launched. Since online SCR gains can be large (particularly at the upper-level and in academic and quantitative subject matters), since our experimental design is robust (course twins are nearly equivalent and the institution’s assets were frozen at the time of this study), and since BCOU student participation in online course versions is not linked to demographic characteristics associated with SCRs, we suggest that BCOU online course delivery features themselves helped students achieve high SCRs.
If online course delivery features per se help independent study students succeed (rather than being the result of students self-selecting themselves into online delivery according to ability,) it is difficult for the institution to justify maintaining twin-course versions to protect access to courses for a traditional audience. If on the one hand SCR gains online were the result of students re-distributing themselves among course versions according to academic ability, overall institutional SCR statistics would represent a zero-sum game and offering twin online and off-line course versions would make sense because this provides traditional audiences with more choice and more time to become familiar with, and adopt online course delivery. If on the other hand the online delivery features themselves help students achieve high SCRs, there exists an ethical imperative (1) To inform students of the online advantage so students can make well informed course selection decisions, and (2) For the institution to strive to move away from ‘course twinning’ toward online delivery.
Since the BCOU has invested considerable assets in developing a twin online and off-line platform however, a mixed virtualization strategy would make sense on an interim basis. “Online delivery offerings only” should be considered in consultation with faculty and students, particularly where SCRs are close to BC system statistics and online benefits are substantial (3rd and 4th year courses in quantitative disciplines for example.) On the other hand twin off-line and online delivery could be maintained where SCRs are similar and close to BC system performance statistics, if the cost-benefits of supporting two course versions warrant, or in areas of special needs. In the ABE programming area where both online and off-line SCRs are considerably lower than BC system performance statistics, new delivery approaches should be contemplated, such as tutor-led paced delivery.
Other institutions that make extensive use of independent study delivery should review their completion rate statistics in the light of this study and research the suitability of Subject Matter and Course Level classifications to optimize their course delivery platforms.
In closing, let us review BCOU independent study course delivery features to help identify which ones might have contributed to high online SCRs. Although this study was not designed to identify benefits among alternate online delivery features, BCOU’s off-line and online delivery models are the product of a rigorous instructional development process and off-line and online course versions exhibit consistent distinguishing features. The main difference between off-line and online course versions is in student-instructor and student-student communication that is available, and how that is mediated and facilitated. Off-line courses do not provide group communications and are based primarily on e-mail tutor support (the use of telephone support has become relatively infrequent). It is a reactive communications model that requires the student to initiate communication around specific problems and issues. In the online courses, students are provided with an online learning environment that integrates the various types of instructional and administrative communication. Through the learning management system (First Class or WebCT) students have direct access to library and administrative support, to the tutor, and to student discussions and conferences moderated by a tutor (students, staff, and tutors are a couple of ‘clicks’ away.) Tutors are more proactively involved in assisting students. This suggests that the integrated learning environment of the online course model yields substantial benefits to credit courses independent study students, particularly at higher levels and in more quantitative subject matters. Supplementing print-based independent study course materials with e-mail tutor support (an approach that we refer to as off-line in this study but that is often considered as online delivery by some institutions) may improve student success over traditional print-based delivery (with tutor telephone support) however this study suggests that SCRs will be even higher if students are provided with an integrated online learning environment.
Acknowledgements
Susan Power, Trevor Chandler, and Mike Looney kindly reviewed drafts of this document. Martin Petter also provided useful information about the concept of stable enrolment. I thank OLA CEO Bill Harlan who supported this project, and TRU-OL COO Colin James who encouraged me to present these findings at the 2006 CADE conference in Montreal. Finally, I thank Editor Mark Bullen and two anonymous reviewers for their thorough reviews.
Discussion
Performance of BCOU Independent Study Online Courses Relative to Their Off-line Counterparts and the BC System
To reach a 1997 goal of increasing completion rates to British Columbia system levels, the BCOU adopted a virtualization strategy based primarily on twinning existing off-line independent courses for online delivery. The BCOU anticipated that re-purposing existing off-line courses online would allow it (a) to establish a strong online presence, rapidly and cost-effectively, and (b) to increase student success while protecting traditional off-line enrolment. Since BCOU assets (staff, curriculum, resources) were frozen shortly thereafter (from 2002 to April 1, 2005) in preparation for its acquisition by another educational institution, and since the BCOU offered a large number of equivalent courses in online and off-line versions at that time, the BCOU represents a ‘natural’ experiment to assess the impact of online delivery on independent study course completion rates. To that effect, we benchmarked BCOU’s online and off-line course completion rates for the academic years 2002/2003 through 2004/2005 using a standard method (Giguère, 2007).
The benchmarking method consists in regressing up to 16 institutional factors on successful completion rate (SCR) data organized according to unique course delivery method and tutor combinations, for each of the academic years 2002/2003 through 2004/2005. The regression method accounted for over 42% of the total variation. Because Course Level and Subject Matter classification loaded in the early steps of regression before tutor specific or course specific factors came into play, it was possible to use Course Level and Subject Matter factors to benchmark online and off-line course completion rates congruently and effectively. The results indicate that the benefits of BCOU’s virtualization has two components, an off-line SCRB increase over time that is possibly due to increased e-mail usage in off-line courses over time (author, 2007), and high online SCRBs relative to off-line SCRBs (this article). Online SCR gains were strong and followed a consistent pattern: they were particularly large for upper-level courses and for courses in academic and quantitative disciplines (SCRB gains of up to 21.2%; Table 6).
By comparing our results and the author’s (2007) to BC system performance statistics provided by course level early during the BCOU embarked on its course twinning virtualization strategy (British Columbia 2000a, 2000b), we can check on BCOU’s 1997 goal to increase course completion rates to BC system levels (Table 10.)
References
Beatty-Guenter, P. (2001). Distance Education: Does Access Override Success? Canadian Institutional Research and Planning Association 2001 Conference, Victoria.
Black, D. (1995). Consortium Student Survey. Burnaby: Open Learning Agency.
Black, D. (1996). Student Services’ Quality of Telephone Service Survey 1996. Burnaby: Open Learning Agency.
Black, D., Bischoff, P., & Bates, A.W. (1994). Further Information on Independent Distance learners at the OLA: Results from the 1994 Omnibus Survey. Burnaby: Open Learning Agency.
Butler, M., & Bischoff, P. (1999). 1999 Student Evaluation Survey: Findings and Recommendations. Open Learning Agency: Burnaby.
Card, K.A., & Horton, L. (1998). Fostering Collaborative Learning among Students Taking Higher Education Administrative Courses Using CMC. ASHE Annual Meeting Paper, South Dakota.
Carr, S. (2000). As distance education comes of age, the challenge is keeping the students. Chronicles of Higher Education, 46(23).
Eliesen, S., & Black, D. (1997). OU/OC Completion Rate Study. Open Learning Agency: Burnaby.
Eliesen, S., & Black, D. (1998). A comparison of course completion rates. Open Learning Agency: Burnaby.
Eliesen, S., & Black, D. (1998). A comparison of on-line and offline course completion rates at the Open University and Open College. Open Learning Agency: Burnaby.
Fredda, J. V. (2000a). Comparison of Selected Student Outcomes for Internet- and Campus-based Instruction. Nova Southeastern University: Fort Lauderdale.
Fredda, J. V. (2000b). Comparison of Selected Student Outcomes for Internet- and Campus-based Instruction at the Wayne Huizenga Graduate School of Business and Entrepreneurship. Nova Southeastern University: Fort Lauderdale.
Fredda, J. V. (2000c). Comparison of Selected Student Outcomes for Internet- and Campus-based Instruction at the Fischlet Graduate School of Education and Human Services. Nova Southeastern University: Fort Lauderdale.
Friesen, N., Breffitt, J., Lefebvre, J., & D. Burgess. (2007). Structured Content for Open Learning. ED-Media World Conference on Educational Multimedia, Hypermedia & Telecommunications, Vancouver, Canada.
Giguère, L. (2002). Profile of BC transfer students registered with the B.C. Open University. B.C. Council on Admissions and Transfer: Vancouver.
Giguère, L. (2007). Benchmarking course completion rates, a method with an example from the British Columbia Open University. Journal of Distance Education, 22(1), 73-86
Holmberg, B. (1995). Theory and practice of distance education. Routledge: London.
Howell, S. L., Laws, R. D., & Lindsay, N. K. (2004). Reevaluating course completion in distance education. The Quarterly Review of Distance Education, 5: 243-252.
Johnson, S. D., Aragon, S. R., Shaik, N., & Palma-Rivas, N. (2000). Comparative analysis of learner satisfaction and learning outcomes in online and face-to-face learning environments. Journal of Interactive Learning Research, 11(1): 29-49.
Keegan, D. (1986). The foundations of distance education. Croom Helm: London.
Klassen, P., Maxwell, J., & Norman, S. (1999). Structured information and course development: An SGML/XML framework for open learning. Open Learning Agency: Vancouver.
Machtmes, K., & Asher, J. W. (2000). A meta-analysis of the effectiveness of telecourses in distance education. The American Journal of Distance Education, 14(1): 27- 46.
McDonald, J., & Gibson, C.C. (1998). Interpersonal dynamics and group development in computer conferencing. American Journal of Distance Education, 12(1): 7-25.
Ministry of Advanced Education, Training and Technology, British Columbia. (2000a). 1998 Performance Report, British Columbia’s College, Institute and Agency System. Victoria: Author.
Ministry of Advanced Education, Training and Technology, British Columbia. (2000b). Open Learning Agency 1998 Performance Report, British Columbia’s College, Institute and Agency System. Victoria: Author.
Munro, P.J. (1991). Presence at a distance. Ph.D. dissertation. University of British Columbia: Vancouver.
Open Learning Agency. (1997). Ends policies: 1997 to 2000. Open Learning Agency: Burnaby.
Paul, R.H. (1986). Access to failure? The challenge of open education at Athabasca University. Community Services CATALYST, 16: 18-22.
Paul, R.H. (1990). Open learning and open management. Kogan Page: New York.
Porter, D. (2001). Object lessons from the web: Implications for Instructional development. In G. Farrell (Ed.). The Commonwealth of Learning: The changing faces of virtual education (pp. 47-63). The Commonwealth of Learning: Vancouver, BC.
Rosberg, W.H. (1997). American government: An introduction using MicroCase with distance learners. Kirkwood Community College: Cedar Rapids.
Russell, T. L. (1999). The no significant difference phenomenon: A comparative research annotated bibliography on technology for distance education. Office of Instructional Telecommunications, Chapel Hill, NC: North Carolina University.
Sener, J. (1996). Delivering an A. S. Engineering degree program through home study distance education. Northern Virginia Community College: Annandale.
Woodley, A., & M. Parlett. (1983). Student dropout. Teaching at a Distance, 24: 2-23.
Zajkowski, M.E. (1997). Price and persistence in distance education. The Journal of Open and Distance Learning, 12: 12-23.
Dr. Louis Giguère is retired and is a part-time consultant on formative evaluations. E-mail: louisgiguere@shaw.ca
PID: http://hdl.handle.net/10515/sy5dz03f5
This work is licensed under a Creative Commons Attribution 3.0 License.
 |