Email this article April – 2012
Conceptual Framework for Parametrically Measuring the Desirability of Open Educational Resources using D-Index
 |
 |
 |
Ishan Sudeera Abeywardena and Choy Yoong Tham
Wawasan Open University, Malaysia
S. Raviraja
University of Malaya, Malaysia
Abstract
Open educational resources (OER) are a global phenomenon that is fast gaining credibility in many academic circles as a possible solution for bridging the knowledge divide. With increased funding and advocacy from governmental and nongovernmental organisations paired with generous philanthropy, many OER repositories, which host a vast array of resources, have mushroomed over the years. As the inkling towards an open approach to education grows, many academics are contributing to these OER repositories, making them expand exponentially in volume. However, despite the volume of available OER, the uptake of the use and reuse of OER still remains slow. One of the major limitations inhibiting the wider adoption of OER is the inability of current search mechanisms to effectively locate OER that are most suitable for use and reuse within a given scenario. This is mainly due to the lack of a parametric measure that could be used by search technologies to autonomously identify desirable resources. As a possible solution to this limitation, this concept paper introduces a parametric measure of desirability of OER named the D-index, which can aid search mechanisms in better identifying resources suitable for use and reuse.
Keywords: Open educational resources; OER; desirability of OER; locating suitable OER; use and reuse of OER; D-index
Introduction
Open educational resources (OER) are fast becoming a global phenomenon, which provides hope for bridging the knowledge divide among the masses (Geith & Vignare, 2008). With increased funding and advocacy by governmental and nongovernmental organisations buttressed by generous philanthropy, many OER repositories boasting a large volume of quality resources have mushroomed over the years. With the movement gaining credibility among many an academic community and with the drive toward opening up knowledge for the benefit of the less fortunate taking centre stage (Johnstone, 2005), these repositories have grown rich in knowledge. However, this has in turn given rise to the new challenge of locating resources suitable for use and reuse from the large number of disconnected and disparate repositories available around the globe (Geser, 2007).
As discussed by Hilton, Wiley, Stein, and Johnson (2010) the use and reuse of an OER depends on two factors: the permission and the technologies needed. The authors introduce the four Rs of openness and the ALMS analysis, which can be used to effectively gauge these factors for identifying the most suitable OER for use and reuse. However, at present, all of the three types of OER repositories, which include content OER repositories, portal OER repositories, and content and portal OER repositories (McGreal, 2010), consider only the relevance of a resource to the search query when locating internal and external resources. Thus, the rank of the search result is not a direct indicator of the suitability of a resource as it does not take into consideration the permission nor the technologies needed to successfully use and reuse. This challenge is further heightened by the common use of OER formats such as PDF, which renders resources useless with respect to reuse (Baraniuk, 2007), and the inability of average users to use the available technological tools to remix the resources (Petrides, Nguyen, Jimes, & Karaglani, 2008). Additionally, as resources are constantly added to these repositories (Dholakia, King, & Baraniuk, 2006), a static method of defining the suitability for use and reuse within the metadata becomes an impossible task.
As a possible solution to this issue, this paper introduces the concept of desirability of a resource, which parametrically takes into consideration (i) the level of openness with respect to the copyright license, (ii) the level of access with respect to technologies, and (iii) the relevance with respect to search rank. The desirability of an OER is then expressed as the D-index which allows search mechanisms as well as users to make informed decisions with respect to the most desirable OER for their needs.
Desirability of an OER
Rationale
In the academic community, the perceived quality of an academic publication or a resource is largely governed by peer review. However, with the present day influx of research publications being made available online, the peer-review mechanism becomes inefficient as not all the experts can review all the publications. As such, an alternative method of measuring the quality of a publication or a resource is needed. According to Buela-Casal and Zych (2010),
If an article receives a citation it means it has been used by the authors who cite it and as a result, the higher the number of the citations the more utilized the article. It seems to be an evidence of the recognition and the acceptance of the work by other investigators who use it as a support for their own work.
Therefore, at present the number of citations received is widely accepted as an indication of the perceived quality of an academic publication or resource.
As the styles of citation for academic publications are very well established, search mechanisms such as Google scholar (see http://scholar.google.com) have a usable parametric measure for providing an indication of how useful a publication would be for one’s academic research. Although there are established styles of citation and attribution for OER as well, these styles are not standardised or widely practiced when using, reusing, remixing, and redistributing OER. As such, it is extremely difficult for a search mechanism to autonomously identify the number of citations or the number of attributions received by a particular OER material. This issue is further amplified as not all the OER repositories available over the Internet are searched and indexed by popular search mechanisms. Providing potential solutions to this issue are systems such as AnnotatEd (Farzan & Brusilovsky, 2006), which uses web-based annotations, use of brand reputation of a repository as an indication of quality, allowing users to review resources using set scales (Hylén, 2005), and the “popularity” in the Connexions repository, which is measured as percentile rank of page views/day over all time. Despite these very specific methodologies, there is still no generic methodology available at present to enable search mechanisms to autonomously gauge the usefulness of an OER for one’s teaching and learning needs.
Definition
The usefulness of an OER for a particular teaching or learning need can only be accurately assessed by reading through the content of the resource. As this is quite a subjective exercise due to one’s needs differing from another’s, it is extremely difficult for a software-based search mechanism to provide any indication of this to a user. This aspect of use and reuse of OER will remain a human function regardless of the improvements in technology. When considering the use and reuse of an OER, there are other aspects of a resource that are fundamental to the usefulness of that particular resource and can be parametrically identified by a software-based mechanism. The first aspect is whether a resource is relevant to a user’s needs. This can be assessed by the search ranking of a resource when searched for with a search mechanism. The search mechanism will compare the title, description, keywords, and sometimes the content of the material to find the best match for the search query. The second aspect is whether the resource is open enough for using, reusing, remixing, and redistributing. This becomes important depending on what the user wants to accomplish with the resource. The third aspect is the accessibility of the resource with respect to technology. If the user cannot easily use, reuse, and remix a resource with available technology, the resource becomes less useful. Therefore, the usefulness of an OER with respect to (i) the level of openness, (ii) the level of access, and (iii) the relevance can be defined as the desirability of an OER, indicating how desirable it is for use and reuse for one’s needs. Within the requirement of being able to use and reuse a particular OER, these three parameters can be defined as follows:
- level of openness, the permission to use and reuse the resource;
- level of access, the technical keys required to unlock the resource; and
- relevance, the level of match between the resource and the needs of the user.
As each of these mutually exclusive parameters are directly proportionate to the desirability of an OER, the desirability can be expressed as a three-dimensional measure as shown in Figure 1.
The Scales
In order to parametrically calculate the desirability of an OER, each of the parameters discussed above needs to be given a numeric value based on a set scale. These scales can be defined in the following ways.
The level of openness can be defined using the four Rs of openness (Hilton, Wiley, Stein, & Johnson, 2010) as shown in Table 1. The four Rs stand for reuse, the ability to use all or part of a work for one’s own purposes; redistribute, the ability to share one’s work with others; revise, the ability to adapt, modify, translate, or change the form of a work; and remix, the ability to combine resources to make new resources. The values 1 to 4 were assigned to the four Rs where 1 corresponds to the lowest level of openness and 4 corresponds to the highest level.
The level of access was defined on a scale of 1 to 16 using the ALMS analysis (Hilton, Wiley, Stein, & Johnson, 2010), which identifies the technical requirements for localisation of an OER with respect to access to editing tools, level of expertise required to revise or remix, ability to meaningfully edit, and source-file access. As shown in Table 2, the value 1 corresponds to the lowest accessibility and value 16 to the highest accessibility.
The relevance of a resource to a particular search query can be measured using the rank of the search results. According to Vaughan (2004) users will only consider the top ten ranked results for a particular search as the most relevant. Vaughan further suggests that users will ignore the results below the top 30. Based on this premise, the scale for the relevance was defined as shown in Table 3, where the value 1 is the least relevant and value 4 is the most relevant.
Calculation
Based on the scales, the desirability of an OER can then be defined as the volume of the cuboid, as shown in Figure 2, calculated using the following formula.
desirability = level of access x level of openness x relevance
As a result, the desirability becomes directly proportionate to the volume of the cuboid.
By normalising the values indicated in Table 1, Table 2, and Table 3 to make the scales uniform for the calculation, the D-index of an OER can be calculated using the following formula.
D-index = (level of access x level of openness x relevance) / 256
Based on the above calculation, a resource becomes more desirable as the D-index increases on a scale of 0 to 1, where 0 is the least desirable and 1 is the most desirable.
Verification of Concept
The most commonly used method for locating OER is to use a generic search mechanism such as Google or to use a search mechanism specific to an OER repository such as Connexions (see http://cnx.org/) or Wikieducator (see http://wikieducator.org). However, both of these types of search mechanisms only consider the relevance of the resource either by matching the title and description or the keywords to the search query provided by the user. Therefore, the resources returned as the top search results might not always be the most desirable for use and reuse in a given scenario as they might be less open or less accessible. The D-index is specifically designed to overcome this limitation by taking into consideration the openness and the accessibility of an OER in addition to the relevance to the search query. When applying the D-index to an OER repository, the level of access, discussed in Table 2, needs to be implemented using the file formats of the OER, where their features are mapped against the ALMS. The level of openness, based on the four Rs discussed in Table 1, needs to be measured using the copyright licensing scheme under which the resource was released. The de facto scheme used in most repositories is the Creative Commons (CC) (see http://creativecommons.org/) licensing scheme, which has six derivations based on the level of openness. However, other specific licensing schemes such as the GNU Free Documentation License (see http://www.gnu.org/copyleft/fdl.html) can also be used for this purpose as long as they can be categorised into the four levels of openness constituting the desirability. Table 4 maps the six CC licences to the four Rs of openness. However, it should be noted that the level of openness of the CC licenses starts at the redistribute level.
Methodology
To verify the accuracy of the proposed D-index, experiments were carried out in three widely used OER repositories: OER Commons (see http://www.oercommons.org), Jorum (see http://jorum.ac.uk/), and MERLOT (see http://www.merlot.org/). These repositories were selected for the experiments due to (i) the repositories providing users with native search mechanisms to locate OER available within the repository as well as hosted outside and (ii) the variety of OER available through them in different levels of openness and access. Each repository was searched using the term calculus to locate OER on the topic of calculus in mathematics. The term calculus was intentionally selected for these experiments due to the large number of OER written and made available on the topic. Only the top 40 search results from each repository, returned based on relevance, were considered in the experiments as the users tend to ignore results below the rank of 30 (Vaughan, 2004).
Calculation of the D-index
To demonstrate how the D-index was calculated for each search result, a general search was conducted on the OER Commons repository for the term calculus using its native search mechanism. Out of the 165 resources returned as results, three resources at the postsecondary level with different search rank were chosen for comparison as shown in Table 5.
The search rank, licence, and the file type of each resource in Table 5 was then compared with Table 3, Table 4, and Table 2 respectively to identify the parameters required to calculate the D-index as shown in Table 6.
Looking at Table 6 we can see that the search mechanism has ordered the results according to the relevance where resource A is the most relevant. However, resource A is less open and less accessible when compared with resource B. Table 7 shows how the results would be reorganised when the D-index is applied to the same search results.
From the results in Table 7, it can be seen that resource B would be the most desirable OER for use and reuse due to its level of openness and access even though resource A was the most relevant.
Experiment Results
Table 8, Table 10, and Table 12 show the top 10 results returned by the native search mechanisms of MERLOT, JORUM, and OER Commons respectively for the keyword calculus. Table 9, Table 11, and Table 13 show the top 10 results when the D-index is applied to the search results returned by MERLOT, JORUM, and OER Commons respectively.
Discussion
By comparing Table 8 and Table 9, which show the search results returned by MERLOT, it can be seen that the original top 10 search results (Table 8) only contain resources that are released under the CC BY-NC-SA license. This license significantly restricts the user’s freedom with respect to the four Rs. Also six of 10 resources returned are in PDF format, which make them difficult to reuse and remix. It must also be noted that the resource ranked as number 10 is a protected resource, which requires a specific username and password to access. Looking at Table 9 where the results are reranked according to the D-index, it can be seen that eight of 10 resources are in HTML/text formats, which are the most accessible in terms of reuse. Four of 10 resources are available under the CC BY licence, which make them the most open resources in the list. Similarly, by comparing Table 10 and Table 11, we can see that the use of the D-index has reranked the top 10 results so that the most accessible resources are ranked at the top instead of resources that use proprietary software applications. The video resources returned were given an accessibility value of 12 according to the ALMS, where access to editing tools = high; level of expertise required to revise or remix = high; meaningfully editable = yes; and source-file access = yes.
Analysing Table 12 it can be seen that four of 10 results returned by the OER Commons search mechanism are copyright protected. As such these cannot be considered as OER and are the least useful for a user who is searching for open material. A value of 0 for openness was assigned to these resources during the D-index calculation. Furthermore, five of the top 10 results returned by the OER Commons search mechanism were in PDF format. Looking at Table 13, it can be seen that the application of the D-index has reranked the resources to provide eight of 10 HTML/text resources. Also the proprietary content has been replaced with open content released under the CC BY and CC BY-NC-SA licenses. The third-ranked resource which is released under the GNU Free Documentation License was assigned a value of 4 for openness during the calculation of the D-index.
By referring to the above results from the experiments conducted on three widely used OER repositories, it can be concluded that the application of the D-index would greatly improve the effectiveness of the search with respect to locating the most suitable resources for use and reuse.
Application and Limitations
The D-index can be incorporated into any search mechanism of an OER repository provided that the resources in the repository are appropriately tagged with the necessary metadata, such as title, description, keywords, copyright license, and file type. Many OER repositories now require authors to define the basic metadata, such as the title, description, keywords, and copyright license. As such, the use of these parameters to gauge the values for relevance and openness becomes an easier task. However, gauging the access parameter which uses the file type of the OER becomes a much more challenging task as some resources consist of multiple files of multiple formats. This can be rectified by breaking a collection of OER into individual learning objects, which allows software applications to determine the file type of the individual OER.
A couple of practical limitations can also be identified with respect to the implementation of the D-index in OER repositories. One of these limitations is that the desirability becomes one dimensional due to the copyright license and the file format being fixed in repositories such as Connexions or Wikieducator. As a result, the D-index becomes only a function of the relevance parameter which does not add much value to the existing search mechanism. Therefore, the D-index is best suited for use in portal repositories/content and portal repositories, such as the OER Commons, MERLOT, and JORUM, which have a wide variety of resources of different file types released under various copyright licenses. It will also be quite effective when used with search mechanisms which query multiple repositories to identify resources.
The other practical limitation is the subjectivity of the search algorithms used by the various native search mechanisms, which results in disparity of the search rank. In turn, this disparity results in the relevance parameter becoming a function of the effectiveness of the search algorithm.
Conclusion and Future Work
Open educational resources (OER) are fast becoming accepted sources of knowledge for teachers and learners around the globe. This is especially true in the case of open distance learning (ODL) institutions where the teaching and learning philosophy is based on open access to education. With the recent developments in technology as well as the establishment of many high quality OER repositories freely available online, the use and reuse of OER should have become mainstream practice. However, as it stands, the use and reuse of OER are still inhibited by a number of technological, social, and economic factors. One of the major technological limitations dampening the use and reuse of OER is the inability to effectively locate useful resources for specific teaching and learning needs from the variety of disconnected and disparate repositories available. This gives rise to the challenge of identifying a parametric measure of the usefulness of an OER, which will enable users to effectively identify suitable resources without reading through countless unsuitable ones.
The concept of desirability of an OER introduced in this paper attempts to lessen the pain of OER users with respect to identifying resources that are relevant, open, and accessible for one’s particular needs. Currently, users who search for OER in specific repositories use search mechanisms native to the repository to identify relevant resources. Depending on the algorithms used by the native search mechanisms, the search query will be compared against the metadata of a resource such as title, description, and keywords to provide a list of resources which might be of relevance. However, these search mechanisms do not take into consideration the level of openness or the technological skills required with respect to using, reusing, remixing, and redistributing a resource. The D-index is an attempt to factor in the openness and accessibility in addition to the relevance in order to provide OER users a useful set of search results which are appropriate to their needs.
The D-index can be incorporated into any OER repository provided that the necessary metadata for calculation are available. It is most effective when used in portal repositories/content and portal repositories which search multiple disconnected OER repositories to locate relevant material. The greatest benefit of the D-index to teachers and learners is its ability to locate and list the most desirable OER for use and reuse from the numerous combinations of relevance, openness, and access under which OER are released. The authors are currently working on incorporating the D-index into an artificial intelligence (AI) based text mining system named OERScout which is used to cluster OER available in all the disconnected repositories based on autonomously identified keywords. The use of the D-index in this clustering process will enable search mechanisms to effectively locate OER which are most desirable for use and reuse.
Acknowledgements
Ishan Sudeera Abeywardena acknowledges the support provided by University of Malaya, where he is currently pursuing his doctoral research in Computer Science, and Wawasan Open University, where he is currently employed.
This research project is funded through the Grant (# 102791) generously made by the International Research Development Centre (IDRC) of Canada through an umbrella study on Openness and Quality in Asian Distance Education.
References
Baraniuk, R. G. (2007). Challenges and opportunities for the open education movement: A Connexions case study. In T. Iiyoshi & M. S. V. Kumar (Eds.), Opening up education – The collective advancement of education through open technology, open content, and open knowledge. Cambridge, MA: Massachusetts Institute of Technology Press. Retrieved from http://citadel.cnx.rice.edu:8180/risa/docs/presskit/cnxbrochuresposter/baraniuk-MIT-press-chapter-oct07.pdf.
Buela-Casal, G., & Zych, I. (2010). Analysis of the relationship between the number of citations and the quality evaluated by experts in psychology journals. Psicothema, 22(2), 270-276.
Dholakia, U. M., King, W. J., & Baraniuk, R. (2006). What makes an open education program sustainable? The case of Connexions. Retrieved from http://www.agri-outlook.org/dataoecd/3/6/36781781.pdf.
Farzan, R., & Brusilovsky, P. (2006). AnnotatEd: A social navigation and annotation service for web-based educational resources. Proceedings: E-Learn 2006–World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, Honolulu, Hawaii. Retrieved from http://www2.sis.pitt.edu/~peterb/papers/NRHM-Final-AnnotatEd.pdf
Geith, C., & Vignare, K. (2008). Access to education with online learning and open educational resources: Can they close the gap? Journal of Asynchronous Learning Networks, 12(1), 105-126.
Geser, G. (2007). Open educational practices and resources - OLCOS Roadmap 2012. Open Learning Content Observatory Services. Salzburg, Austria. 2007. Retrieved from http://www.olcos.org/cms/upload/docs/olcos_roadmap.pdf.
Hilton, J., Wiley, D., Stein, J., & Johnson, A. (2010). The four R‘s of openness and ALMS Analysis: Frameworks for open educational resources. Open Learning: The Journal of Open and Distance Learning, 25(1), 37-44.
Johnstone, S. (2005). Open educational resources serve the world. Educause Quarterly, 28(3).
McGreal, R. (2010). Open educational resource repositories: An analysis. Proceedings: The 3rd Annual Forum on e-Learning Excellence, Dubai, UAe. Retrieved from http://elexforum.hbmeu.ac.ae/Proceeding/PDF/Open%20Educational%20Resource.pdf.
Petrides, L., Nguyen, L., Jimes, C., & Karaglani, A. (2008). Open educational resources: inquiring into author use and reuse. International Journal of Technology Enhanced Learning, 1(1/2).
Vaughan, L. (2004). New measurements for search engine evaluation proposed and tested. Information Processing and Management 40, 677–691.
