Email this article April - 2004
Technical Evaluation Report
30. Vendor-Assisted Evaluation of a Learning Management System
Abstract
A product evaluation was conducted of Desire2Learn, an online learning management system gaining popularity in educational institutions. Since an online trial version of the software was not available for evaluation, an inspection meeting with the vendor was arranged. This provided the evaluation team with an opportunity to examine the precautions necessary for ensuring the objectivity of a product evaluation based on vendor-supplied information. The report outlines the team’s use of evaluator-driven enquiry and a triangulated approach to information checking. The Desire2Learn product is assessed as a flexible and useful addition to the range of learning management system options.
Introduction
A previous report in this series (#25) adopted the convention of grouping course and content management systems under the heading Learning Management System (LMS) to avoid confusion between CMS systems. Proliferation of such acronyms is just one example of the types of confusion currently reigning in this rapidly advancing field. Report #10 in the series described another source of confusion, relating to the lack of objectivity that can arise when software is evaluated with the assistance of its developer/vendor. Often there may be no choice in the matter, when a trial version of the product is not available for inspection. The previous report listed the precautions required when the evaluator is obliged to seek the vendor’s assistance in order to gain access to the product. The current study provides a direct example of this situation, in describing an LMS evaluation conducted via a face-to-face demonstration by the vendor in a Vancouver educational institution. The report discusses the precautions taken by the evaluation team in maintaining its objectivity throughout this process, and the outcome of the study.
Methodology
Desire2Learn (D2L) is an integrated online learning management system (LMS), which provides synchronous and asynchronous interaction between students, teachers, and learning content. Such platforms are usually integrated with administrative databases (e.g., commercial products such as SCT Banner, Peoplesoft, Datatel, and Campus Pipeline).
Information for possible use in evaluating the product was collected in September and October 2003, from varied sources, including:
- Personal access to existing D2L courses
- Interviews with D2L users
- An interview with an independent evaluator of D2L in a post-secondary institution
- Information on the D2L company’s website
- Demonstration of the software by a D2L representative at a Vancouver educational institution
- The vendor’s website provided mission and vision statements, information
about external market standards, general information about the product and
services, a selective demonstration of features, links to reference sites
and adjunct services, and contact information for product representatives.
- As Fahy (2002) points out in Report #10, “it is not reasonable to
expect vendors to supply complete, objective information on their own products.”
Prior to the meeting with the D2L representative, therefore, the evaluators
solicited impartial information about the product from individuals in two
major post-secondary institutions. One of these institutions has used D2L
for three years, and is doing so currently. The other institution is in the
process of assessing whether to transfer to D2L from another LMS platform.
- During the meeting with the D2L representative, the evaluators requested
and received a detailed demonstration of specific features of the software
in order to verify the qualitative and comparative information provided on
the company’s website. During the review, they explored several reference
sites via the representative’s access to them.
- While the company’s website provides technical specifications to externally recognized standards, the face-to-face meeting allowed the team to learn first-hand about the history and evolving functionality of the product. This meeting also provided insight into the organization and management of the company, pricing structures, product support, and software upgrades. At times the discussion with the representative had to be re-focused on the current applications of the product rather than on future development possibilities.
Prior to the visit to the D2L representative by two of the three-member evaluation team, steps were taken to ensure the depth and objectivity of the information that would be obtained. The caveat emptor recommendations of Report #10, were followed in this connection. In addition to recognizing the need for independent corroboration of the data collected, the evaluators knew they should request the evidence for unsubstantiated claims and should be prepared, if doubtful of any claims, to challenge the vendor to prove them. Objective technical information was sought, where necessary, to replace merely qualitative product descriptions.
Highlights of the information-gathering process were as follows:
The ‘forewarned-is-forearmed’ approach was considered beneficial, and it is gratifying to report the company responded obligingly to the evaluation team’s approach at all times. (In fact, the D2L company’s reputation for a high level of customer service had already reached us by word-of-mouth before the evaluation was conducted.) The D2L representative was open and forthcoming, receptive to feedback and committed to our discussion of educational criteria.
Evaluation of Desire2Learn
The D2L software has a startup cost based on the number of features, the level of integration with existing systems, and the support needs of the institution. The product has the capability to be used as a course-by-course delivery tool, or scaled up to a larger entity such as a province-wide license. Annual licensing costs are based on a flat fee or on a per-student basis. Users are not required to download any software. The company’s servers contain all the software to support their LMS platform with the exception of Java, which is used under the license. This minimal use of third-party software enables the company to give flexible responses to clients’ requirements. The vendor provides access to a support person per 5000 students covered by the license.
The product is essentially asynchronous in its approach to online collaboration; at the time of the evaluation, it does not contain synchronous tools such as audio, whiteboard, instant polling, and co-browsing. Its asynchronous conferencing tools are flexible for both administrators and users. The licensing approach allows administrators to assign roles across multiple organizations to facilitate a consortium approach. Administrators can create an unlimited number of customized user roles, with specific privileges for instructors and students. D2L is XML standard-based, and uses a learning-object repository approach that allows instructors to store content (i.e., pages, graphics and test-item banks, etc.). These objects can be shared across any number of courses or programs. (By contrast, other prominent LMS systems use hard-coded HTML pages uploaded to a specific version of a course.) Administrators can assign students the ability to search for specific course content and discussion threads in the learning-object repository. A context-sensitive help feature provides help on each of the major tools for both developers and students. The software is compliant with leading international course-sharing standards including SCORM and IMS, and meets international standards for access by disabled users. D2L databases can be configured to communicate directly with other institutional databases.
The customizable features of D2L provide administrators with straightforward means of creating conferences for the members of a class or its sub-groups. Students can be assigned access privileges for system-wide text-chat rooms. Once conferences are created for an individual course, new conferences can easily be created for subsequent offerings of the course. Within a specific course offering, administrators can edit or delete threads, and can control the access privileges of users at different levels, including the right to edit and/ or delete one’s own messages. Threaded messages are displayed clearly, and users have control over keyword searches and message sortings by date, topic, and sender. Other options include saving, printing, editing and deleting of postings.
Conclusions
A primary strength of D2L is the breadth and adaptability of its features. It is a fully integrated learning management system that has evidently evolved through close relationships between the software developer and educational institutions. Because D2L is XML standard-based, it is relatively easy to integrate it with other database systems. The product has a high degree of usability. Its course structure and appearance are easy to manipulate, and administrative control over the usage process is broad and easily specified. Owing to its high degree of flexibility and customizability, the product can take many forms. For this reason, the evaluation team recognized the vendor’s preference for demonstrating its product in person, on an individualized basis rather than via a more generalized online demonstration approach
Acknowledgements
The authors thank Ben Moore of the Desire2Learn company for his assistance in providing information about this software product; also Bryan Fair, evaluator of Desire2Learn at the British Columbia Institute of Technology.
The next report in the series describes the use of synchronous audio methods in a cross-cultural setting.
N.B. Owing to the speed with which web addresses become outdated, online references are not cited in this report. They are available, together with updates to the current report, at the Athabasca University software evaluation site: http://cde.athabascau.ca/softeval/. Italicised product names in this report are assumed to be registered trademarks.
JPB, Series Editor, Technical Evaluation Reports
