Email this article 
Post a Comment June – 2009
Editorial
The Ratings Game
Terry Anderson
Editor, IRRODL
This issue is notable as it is the largest single issue ever published by IRRODL! The issue contains fourteen peer-reviewed research articles, two technical reports, and links to five recordings and Powerpoint slides from research presentations to IRRODL’s sister organization, the Canadian Institute for Distance Education Research (CIDER). It also contains two articles formatted for mobile devices (EPUB), and we welcome feedback, particularly from Amazon Kindle and Stanza users.
As usual, the subject and research methods employed in the articles are heterogeneous, though all content has a focus on distance education. Ten of the articles relate to innovations in teaching and learning in distance contexts. Issues covered include synchronous technologies, multimedia training in industry contexts, web conferencing in high schools, strategies to create online group collaboration, and effective and ineffective instructor behaviours We also have two very interesting conceptual pieces, one on the role of culture in online learning and the other highlighting the effect of transparency on cooperative learning. The issue begins with two excellent articles relating directly to distance education research. The first presents the results of a Delhi study in which international experts prioritized distance education research issues and trends. The second is an article that provides a ranking system and the results when it is used to assess computer-mediated learning journals. I confess that I cannot resist bringing to your attention that IRRODL rates as the top-ranked journal amongst 46 contenders!!
The ratings game, while potentially both ego boosting and deflating for editors and authors, has very serious implications in academia. Enormous attention is placed in tenure and promotion hearings on not only the number of publications a scholar produces but also on the quality of the journals in which this work appears. Unfortunately, like many other measures of quality, determining the quality (often referred to as ‘impact factor’) of scholarly journals raises many issues including the validity of the results, the reliability of the measurement scales, and the effect of regional, linguistic, and pecuniary bias. Much work has been done to create valid indexes of journal quality, and Elbeck’s and Mandernach’s (this issue) innovative assessment strategies add to this literature. The digitization of articles and their accessibility on the web (either through subscription to databases of commercial publications or through open access) creates opportunity for new metrics to calculate impact. These include the number of downloads and links from other sources. However, citation of the work by one’s peers remains the most credible measurement of impact.
Thus, I was very pleased to bump into the work of Australian researcher Anne-Wil Harzing and the interesting tool she has developed and released freely for non-commercial use. The downloadable tool Publish or Perish uses the Google Scholar database to aggregate the number of articles and the number of times they have been cited by others. These citation numbers are then used to calculate an ‘impact factor” of both individual authors and journals. The tool also provides a number of algorithms that attempt to resolve the problem of multiple authors and also allow one to restrict publications used in the calculations by date or subject matter. Although Google Scholar’s data collection methods remain clouded in commercial obscurity, Harzing’s tool is at worst very entertaining and at best a very informative means by which researchers can select appropriate journals for publication and quantitatively measure the impact of their own and their colleagues’ work. I won’t give the results away (yet), but please feel free to spend a few entertaining minutes comparing the 12 distance education journals currently publishing or calculate your own and your colleagues’ impact factors!