Evaluation of the Language Portal of Canada (final report)
June 22, 2016
On this page
Main points
- The Language Portal of Canada is a federal initiative undertaken in support of the Government's commitment to promote official languages and enhance the vitality of official-language minority communities. The Language Portal provides Canadians with online access to an assortment of official language resources intended to assist with communicating effectively in English and French, while at the same time providing a platform for Canadian language expertise. The Translation Bureau, a special operating agency of Public Services and Procurement Canada, is responsible for the development and management of the Language Portal of Canada.
- Funding for the Language Portal stems from the Roadmap for Canada's Official Languages 2013 to 2018: Education, Immigration, Communities which is a Government-wide investment in Canada's linguistic duality. As part of the Roadmap funding for the Language Portal, the Translation Bureau is developing and maintaining the Language Portal's content, and making it available online.
- As indicated in the Language Portal's logic model, its objectives are that:
- Canadians have access to quality language resources in both official languages (immediate outcome)
- Canadians benefit from using the Language Portal of Canada and its language resources (intermediate outcome)
- the Language Portal of Canada contributes to improving Canadians' knowledge and use of the two official languages (ultimate outcome)
- Overall, the evaluation found that there is a continued need for the Language Portal based on federal requirements related to official languages, and as evidenced by the increasing number of visits to the Language Portal website. The evaluation also found that the Language Portal aligns with federal government priorities and Public Services and Procurement Canada (PSPC) strategic outcomes, including those of the Translation Bureau. Roles and responsibilities related to delivering the Language Portal align with the Department's mandate, and supports the requirement in the Official Languages Act that every federal institution implement positive measures to advance linguistic duality in Canadian society.
- The evaluation found that the Language Portal is meeting its immediate outcome that Canadians have access to quality language resources in both official languages. The Language Portal Division has been producing and making available hundreds of French and English tools and resources over the last two years of Language Portal funding. The evaluation could not conclude that the Language Portal is meeting its intermediate outcomes of Canadians drawing benefits from using the Language Portal of Canada and its language resources, as the required data was not available during the conduct phase of the evaluation. However, anecdotal evidence suggests that Canadians are drawing benefits from the Language Portal. The intermediate outcome for the Language Portal will be measured and reported on once the planned user satisfaction survey is elaborated and placed on the website (planned for 2016 to 2017) when it is migrated to Canada.ca.
- Program documents indicate that Language Portal spending is in line with planned budgets. Although the evaluation team was unable to benchmark the Language Portal costs against a similar online language website, a comparable website run by the Government of Quebec was examined. Although this organization and website has a different mandate than the Language Portal and larger budget, both websites had similar administrative expenses ratios at roughly 40% in 2014 to 2015.
Introduction
1. This report presents the results of the Evaluation of the Language Portal of Canada, which is included in the Department's 2015 to 2016 Program Alignment Architecture (PAA) as sub-program 1.6.1 (Terminology Standardization Program). The Language Portal first appeared in the Public Works and Government Services Canada's Program Alignment Architecture in 2009. The Language Portal is managed and maintained by the Translation Bureau's Language Portal Division.
2. The Language Portal is designed to provide Canadians with a single point of access, free of charge, to a wide range of language tools and resources. Funding for the Language Portal stems from a government-wide investment in Canada's linguistic duality of $16 million over five years as part of a government-wide investment of $1.15 billion through the Roadmap for Canada's Official Languages 2013 to 2018: education, immigration, communities. A total of 14 organizations received funding as part of the Roadmap. A condition of the Roadmap funding is that each department must conduct an evaluation of its respective initiative and provide it to the Department of Canadian Heritage by June 30, 2016.
3. This evaluation was included in the 2015 to 2018 Risk-Based Audit and Evaluation Plan. The evaluation provides an assessment of the relevance and the performance of the Program for the period April 2013 to October 2015. Planning and research took place between August and December 2015, in accordance with the Standard on Evaluation for the Government of Canada and in conformity with the PSPC Policy on Evaluation. More information on the approach, methodologies and limitations of this evaluation can be found in Appendix A: About the evaluation.
About the program
4. Canada is a bilingual country; the Official Languages Act, which was enacted in September 1969, establishes English and French as the official languages of Canada and is intended to ensure that each language has equality of status as to their use in all institutions of Parliament and the Government of Canada. The Official Languages Act also requires that federal institutions take positive measures to promote linguistic duality. The Official Languages Act emerged from the recommendations of the Royal Commission of Inquiry on Bilingualism and Biculturalism. Later, with the ratification of the Constitution Act, 1982, the concept of official languages became enshrined in the Constitution of Canada. Under section 42 of the Constitution Act, the responsibility for encouraging and promoting a coordinated approach to the fulfilment of the commitments of federal institutions was conferred to the Minister of Canadian Heritage.
5. To strengthen its implementation of commitments under the Official Languages Act, the Government created the Action Plan for Official Languages (2003 to 2008), which was directed at building on the foundation for linguistic duality established by the legal framework for official bilingualism. This was followed by the Roadmap for Canada's Linguistic Duality (2008 to 2013), which in turn was followed by the current initiative, the Roadmap for Canada's Official Languages (2013 to 2018). The Roadmap is viewed as one of the many components that make up the Government of Canada's Official Languages Program and serves to reaffirm the Government of Canada's commitment to promote official languages and enhance the vitality of official-language minority communities.
6. PSPC's unique responsibilities and assets as a common service provider, in addition to existing business lines related to official languages, establish the context for its role in advancing Canada's Roadmap agenda. It is within this context that the Language Portal of Canada was launched by the Translation Bureau in 2009 as an initiative under the Roadmap (2008 to 2013).
7. In 2013, the Government of Canada renewed its commitment to promoting official languages by launching the Roadmap for Canada's Official Languages 2013 to 2018: Education, Immigration, Communities. This renewed commitment laid out the Government's official languages strategy for 2013 through to 2018 and included $16 million for Public Services and Procurement Canada to continue to expand the Language Portal over the five year period. This most recent funding of the Language Portal serves to reaffirm its position as a source of Canadian expertise in the area of language, and has allowed the Translation Bureau to expand its content.
8. The Language Portal provides Canadians with access to a broad range of quality language resources in English and in French. It is not a translation tool, but rather a portal that allows single window access to a broad collection of language tools and resources. As part of the 2013 to 2018 funding for the Language Portal, the Translation Bureau expanded the Language Portal's content to include articles, writing tools, headlines, tutorials and quizzes on a variety of subjects (example, grammar and syntax, vocabulary, writing styles, literacy and others). It also made resources previously developed by the Translation Bureau for use within the public service available to the general public via the Internet. These resources included TERMIUM Plus®, ConjugArt, Word Tailoring, HyperGrammar2 and other tools. The Translation Bureau also established partnerships with Canadian language sector stakeholders to provide access to additional information and language resources on the Language Portal (example, articles, links to information and resources on external websites) and developed and launched promotional and marketing campaigns to introduce the Language Portal to Canadians.
9. The Language Portal is designed to support Canadians, students and educators, and the language industry by providing tools and resources that support Canadian capacity to communicate in both official languages. It also aims to highlight Canadian expertise in the area of language.
10. The Language Portal was last evaluated by the Department in 2011, and the findings from the 2011 evaluation were used to inform the 2012 Horizontal Evaluation of the Roadmap which was conducted by the Department of Canadian Heritage. The 2011 evaluation made two recommendations for improving the relevance and performance of the Language Portal. A management action plan was developed by the Translation Bureau to address these recommendations and has since been fully implemented. Full implementation of the management action plan was validated by the Office of Audit and Evaluation in December 2014.
Authority
11. The Department of Public Works and Government Services Act grants PSPC the authority to provide translation and related services to the Government of Canada departments, boards and agencies, and also to provide such other services as the Governor in Council may direct.
12. The Minister of Public Services and Procurement is responsible for the administration of the Translation Bureau Act and associated regulations.
13. In accordance with Part VII of the Official Languages Act amended in 2005, every federal institution is required to ensure that positive measures are taken to implement the Government of Canada's commitment to support linguistic duality and the development of official-language minority communities.
Roles and responsibilities
14. The Language Portal of Canada is located within the Language Portal Division of Corporate Services at the Translation Bureau located in Gatineau, Quebec. Governance of the Language Portal takes place through the existing reporting structure within the Translation Bureau.
Resources
15. In 2014 to 2015, the Language Portal expenditures were $3,094,371; the group is comprised of 16 employees. Roadmap (2013 to 2018) funding for the Language Portal is $16 million over five years.
Logic model
16. A logic model is a visual representation that links a program's activities, outputs and outcomes; provides a systematic and visual method of illustrating the program theory; and shows the logic of how a program is expected to achieve its objectives. It also provides the basis for developing the performance measurement and evaluation strategies, including the evaluation matrix.
17. A logic model for the Program was developed based on a detailed document review, meetings with program managers and interviews with key stakeholders. It was subsequently validated with program staff. The logic model is provided as Exhibit 1: Logic model for the Language Portal of Canada.
Exhibit 1: Logic model for the Language Portal of Canada
Image description of Exhibit 1
The exhibit depicts the Language Portal of Canada. Going from the top down, the model describes the Language Portal of Canada's as PSPC's initiative, and its activities, output, immediate outcome, intermediate outcome, ultimate outcome, and ending with the horizontal strategic outcome for Roadmap for Canada's Linguistic Duality, and the linkages between them.
Public Services and Procurement Canada's Initiative
PSPC's Initiative is identified as the Language Portal of Canada. This is linked to both of the following activities.
Activities
- Develop, maintain and promote the Language Portal of Canada with online language resources.
- Establish agreements with language services partners to enrich the content of the Language Portal of Canada.
Both activities are linked to the following output.
Output
- Bilingual Language Portal of Canada.
The output is linked to the following immediate outcome.
Immediate outcome
- Canadians have access to quality language resources in both official languages.
The immediate outcome is linked to the following intermediate outcome.
Intermediate outcome
- Canadians benefit from using the Language Portal of Canada and its language resources.
The intermediate outcome is linked to the following ultimate outcome.
Ultimate outcome
- The Language Portal of Canada contributes to improving Canadian's knowledge and use of the two official languages.
The ultimate outcome is linked to the following horizontal strategic outcome.
Horizontal strategic outcome for Roadmap for Canada's Linguistic Duality 2013 to 2018
Create conditions in which Canadians can live and thrive in both official languages, and recognize the importance of English and French to Canada's national identity, development and prosperity.
Program activities
18. As per the logic model, there are two main activities associated with the Language Portal. Descriptions for each are found below:
- Develop, maintain and promote the Language Portal of Canada with online language resources: The Translation Bureau's Language Portal Division develops and manages new content in both official languages for the Language Portal, including weekly articles and headlines, Quizzes and Flash Quiz questions, and language articles. They also created a search tool called Le français sans secrets | Gateway to English, which allows users to search simultaneously through a host of language content, in addition to incorporating many Web 2.0 features to the Language Portal
- Establish agreements with language services partners to enrich the content of the Language Portal of Canada: The Translation Bureau established partnerships with 20 (as of August 2015) language sector stakeholders to provide access to additional information and language resources on the Language Portal (example, articles, links to information and resources on external websites). They also developed and launched promotional campaigns to introduce the Language Portal to Canadians. The partners are listed on the Language Portal and include the Canadian Foundation for Cross-Cultural Dialogue, the Canada School of Public Service, and the Language Technologies Research Centre, among others
Evaluation reporting
19. The Deputy Minister for Public Services and Procurement Canada approved the conduct of this evaluation, at the recommendation of the Audit and Evaluation Committee, as part of the 2013 to 2018 Risk-Based Audit and Evaluation Plan. Planning and research took place between August and December 2015, in accordance with the Standard on Evaluation for the Government of Canada and in conformity with the PSPC Policy on Evaluation. More information on the approach, methodologies and limitations of this evaluation can be found in Appendix A: About the evaluation.
20. The Evaluation Directorate completed a risk-based calibration assessment to determine the best approach to conduct this evaluation. The assessment identified the Language Portal of Canada as an appropriate candidate for a streamlined evaluation for the following reasons:
- the Program represents less than 1% of PSPC's annual spending
- the Program has a simple logic model and centralized delivery mechanism
21. As such, the results of the evaluation research and analysis are presented in table format. The tables present the findings and conclusions by evaluation issue, while citing the indicators and evidence used. This reporting format meets the requirements for evaluation reporting set out in Section 6.4.1a (i to ix) of the Treasury Board Standard on Evaluation for the Government of Canada.
Section 1: Relevance
Continued need
To what extent does the Language Portal address a demonstrable need?
| Indicator | Findings | Source | Conclusion |
|---|---|---|---|
| Evidence of a need for increasing or supporting capacity, knowledge, and use of both official languages by Canadians, students and educators, and the language industry | The Language Portal was made available to Canadians in accordance with federal official language priorities identified in the Official Languages Act.
Key objectives of the Official Languages Act relevant to the Language Portal include:
Although there is no specific legislative or policy requirement for PSPC to deliver the Roadmap Initiatives, the Department's delivery of the Language Portal contributes to the objectives of the Official Languages Act. At the end of 2014 to 2015, there were 3,424 content items available on the Language Portal, including writing tools, languages articles and quizzes. This is a 38% increase from the 2,475 content items that were available at the end of 2012 to 2013. By providing web-enabled linguistic tools and resources, it supports federal institutions in discharging their responsibilities to implement positive measures to foster recognition and use of English and French in Canadian society. In addition, the Language Portal provides official language minority communities with language tools and resources in both official languages, announces activities and events that concern them, and provides them with a window so that they, too, can announce activities related to either official language. The Language Portal supports the equal status and use of both official languages by making approximately the same amount of content available in both English and French to Canadians nationwide. For example, in 2014 to 2015, there were 234 content additions to the Language Portal in French and 234 in English. Similarly, hyperlinks to external language sources in 2014 to 2015 reflected a nearly even split, at 56% English and 44% French. This equality in content delivery is reflected in the patterns of use of the Language Portal - in both 2014 to 2015 and 2015 to 2016, approximately 46% of Language Portal users visited French pages while 54% visited English pages. The Language Portal also contributes to the minority language community objectives of the Official Languages Act by providing access to content and links for French language minority users in predominantly English-speaking areas of Canada, and content and links for English language minority users in Quebec. |
Program documents | The Language Portal provides tools and resources to enhance capacity, knowledge, and use of both official languages. |
| Evidence of 'client' use of the Language Portal | Use of the Language Portal has been increasing annually.
According to the 2013 to 2014 Report on Results – Implementation of Section 41 of the Official Languages Act, official language minority communities are among the Language Portal's target groups. Out of the 190 headline articles posted on the Language Portal in 2013 to 2014, 39 were from Official Language Minority Communities. Language Portal content was viewed 2 million times in the combined 2010 and 2011 calendar years. In comparison as shown in Appendix B Table 8, in a similar 24-month period spanning the 2013 to 2014 to 2014 to 2015 fiscal years, the Language Portal was viewed 7.95 million times. This represents an increase of nearly 400%. As noted above, use of the Language Portal, in both 2014 to 2015 and 2015 to 2016, was almost evenly divided between French-language and English-language resources. In each of those fiscal year periods, approximately 46% of Language Portal users visited French pages while 54% visited English pages. A more detailed breakdown of Language Portal users sorted by browser language and geographic location, can be found in the outcome achievement section, below. |
Program documentation | Data indicates that users are increasingly aware of the Language Portal, as visits to the Language Portal are increasing. |
Alignment with Government priorities
Does the Language Portal align with federal government priorities and PSPC strategic outcomes?
| Indicator | Finding | Source | Conclusion |
|---|---|---|---|
| Evidence of the alignment of the Language Portal objectives with federal government priorities and departmental strategic outcomes | In the 2013 Speech from the Throne, the Government of Canada underlined the significance of Canada's bilingualism and cultural diversity in the context of the anticipated activities to commemorate Canada 2017. The Government then announced via the 2013 Budget that the Roadmap for Canada's Official Languages (2013 to 2018) would be renewed. The 2013 Budget specifically underlined the significance of Canada's two official languages as an integral part of Canada's history, culture, and identity as a nation, and the Government's continued commitment to Canada's official language minority communities and to strengthening linguistic duality.
The Roadmap (2013 to 2018) reaffirmed the Government's commitment to promote official languages and enhance the vitality of official language minority communities in three priority sectors: education, immigration and communities. The implementation of this initiative involves several agencies and departments, based on their specific mandate. Considering the broad mandate of PSPC as a common service agency for the Government of Canada, and Translation Bureau's responsibilities for the provision of linguistic services, the PSPC contribution is associated with the first priority sector: Education. The 2015 Speech from the Throne also indicated that the Government would encourage and promote the use of Canada's official languages, and invest in Canada's cultural and creative industries. Consistent with government responsibilities related to official languages (as outlined in the Continued need section, above), the Language Portal is designed to help Canadians communicate in both official languages; disseminate and promote language resources developed in Canada; and, share and highlight Canadian expertise in the area of language. In addition, the Language Portal aligns with PSPC's strategic outcome related to the delivery of "high-quality, central programs and services that ensure sound stewardship on behalf of Canadians and meet the program needs of federal institutions" by providing federal employees with access to high quality linguistic tools and resources. It also aligns with the program outcomes for Linguistic Management and Services and this program's mandate to develop terminology standards to ensure clear, uniform and quality communications within government. The Language Portal supports this objective by providing a single point of access to high-quality, standard language content, accessible to all federal employees. According to the Departmental Performance Report of 2013 to 2014, PSPC continued to act as the federal government's terminology authority through the Language Portal of Canada and TERMIUM®, the Government of Canada's terminology and linguistic data bank. This was reiterated in the PSPC 2015 to 2016 Report on Plans and Priorities which states that the department has committed to continuing to ensure that both Government of Canada employees and Canadians at large have access to quality linguistic tools to help them communicate in both official languages and improve comprehension and the ability to write and speak. |
Program documents | The Language Portal is aligned with federal government priorities and PSPC strategic outcomes. |
Alignment with federal roles and responsibilities
Is the Language Portal consistent with federal roles and responsibilities?
| Indicator | Finding | Source | Conclusion |
|---|---|---|---|
| Evidence of the alignment with federal government's roles and responsibilities in relation to the Language Portal | The Official Languages Act is federal legislation that governs federal government activities and services. As such, the federal government has a responsibility to support its implementation.
The Department of Public Works and Government Services Act requires PSPC to operate as a common service agency for the Government of Canada, and for its activities as a common service agency to be directed mainly toward providing the departments, boards and agencies of the Government of Canada with services in support of their programs. Section 6(i) of the Act refers to PSPC responsibilities associated with translation and related services. Further, the Translation Bureau, through the Translation Bureau Act, is given responsibility for various duties and functions related to Canada's official languages, and is identified as an agency within PSPC. Through the Official Languages Act, the federal government has established roles and responsibilities for federal departments related to the advancement of both official languages. Under the Official Languages Act, each federal institution is responsible for implementing the Act within its area of jurisdiction. In this context, the Language Portal, developed by PSPC as a common service agency, is a tool that helps all federal institutions comply with the objectives of the Official Languages Act. |
Program documents | The federal government's delivery of the Language Portal is aligned with government roles and responsibilities outlined in the Official Languages Act, the Public Works and Government Services Act, and Translation Bureau Act. |
| Evidence of consistency and compliance with government regulations and priorities | The Translation Bureau created the Language Portal of Canada in response to the official language commitments made by the federal government.
The Translation Bureau Regulations require the Translation Bureau to maintain a Terminology Centre to carry out terminological research and prepare and distribute terminological instructions and bulletins and make its services available to departments. In recognition of the importance of this initial work in support of the use of both official languages in federal institutions, the Government approved $34M in additional funding for the Translation Bureau to develop the Language Portal as part of the Roadmap for Canada's Linguistic Duality 2008 to 2013: Acting for the Future. The 2008 to 2013 Roadmap represented an unprecedented investment in Canada's linguistic duality, and together with the current Roadmap for Canada's Official Languages 2013 to 2018: Education, Immigration, Communities, establishes the importance that successive governments have placed on promoting the equality of Canada's official languages. The Language Portal is well-aligned to this Government of Canada priority. The mandate letter for the Minister of Canadian Heritage lists as a priority for that Minister to establish a free, online service for learning and retaining English and French as second languages. The Minister of Public Services and Procurement was not assigned any priorities related to official languages. The Language Portal Division has been in communication with the Department of Canadian Heritage to offer the accomplishments of the PSPC Language Portal to support achievement of the outcomes in the Minister of Canadian Heritage's mandate letter. Discussions will continue in 2016, and the Language Portal could act as a platform for the second language learning tool to be established by the Department of Canadian Heritage. |
Program documents | The Language Portal supports and is consistent with government regulations and priorities related to official languages. |
Conclusion: Relevance
The Language Portal, through the funding provided via the Roadmap for Canada's Official Languages, supports Canadian linguistic duality and expertise in the official languages of Canada by providing free access to Canadian language resources. The volume of resources provided and the usage of those resources demonstrates a continuing need. The Language Portal of Canada is consistent with federal responsibilities as per the legislated requirements of the Official Languages Act, the Translation Bureau Act and the Public Works and Government Services Act.
Section 2: Performance
Outcome achievement
Immediate outcome
To what extent has the Language Portal contributed to language resource availability in both official languages?
| Indicator | Findings | Source | Conclusion |
|---|---|---|---|
| Increase in number of resources/tools made available, by language and type over the evaluation period
and Percentage increase of resources/tools added to the Language Portal, by language and type over the evaluation period |
Since the renewal of the Language Portal funding, the Language Portal Division has increased the amount of content made available on the website.
The overall accomplishments of the Language Portal over the evaluation period have been aggregated in Appendix B Table 8. Below are samples of some of the tools and resources developed over the evaluation period. There were 2,475 tools and resources available on the Language Portal at the end of 2012 to 2013. This number increased to 3,424 (38% increase) by the end of 2014 to 2015. This consisted of over 450 additions to the Language Portal in both 2013 to 2014 and 2014 to 2015. Quizzes represented over 25% of the content increase, with the remainder made up of articles. The volume of new content was developed equally in English and French. Over 200 new links to external content were also added to the Language Portal in each of those fiscal years, representing an increase of approximately 10%. |
Program documents | The Language Portal Division has increased the number of tools and resources available on the Language Portal over the evaluation period in both official languages. |
| Extent to which the Translation Bureau's Language Portal Division has produced and integrated original content into the Language Portal | Accredited language professionals working for the Language Portal Division are responsible for creating 90% of the tools and resources available on the Language Portal. This includes:
|
Interviews | The Language Portal Division has produced a collection of official language tools and resources for publication on the website. |
| Extent to which other partners have contributed to the content of the Language Portal | Content has also been developed for the Language Portal by 288 partners/collaborators and ambassadors representing the remaining 10% of the total content available directly from the website. The collaborators have a mandate to produce articles that reflect their reality, their activities, their accomplishments and their field of work. Some of the organizations affiliated with the collaborators include teachers associations, writers and reviewers' associations, universities and provincial governments. Notable Language Portal contributors include:
|
Interviews | A noteworthy amount of content is produced by partners of the Language Portal. |
| Evidence that the Language Portal has increased its visibility, as per recommendation from the 2011 evaluation | The recommendation from the 2011 evaluation stated that the Translation Bureau "should consider promotional or other activities to increase the visibility of the Language Portal of Canada and awareness of its language tools and resources by all Canadians". The management response stated that the Translation Bureau would develop a Strategic Promotional Plan, by:
Since 2011, the Language Portal Division has engaged a media relations firm to manage its annual search engine based advertising campaign. In describing the process, the Language Portal Division indicated that they identified thousands of keywords and terms that a person might enter in a search engine when looking for a website offering content similar to the Language Portal's. During the advertising campaigns, when users entered one of those keywords or terms into a search engine, an advertisement about the Language Portal would be presented and users would be able to click on it to be directed to the Language Portal website. The number of clicks are monitored and transformed into a metric known as a "click through rate" which shows how often people who see an advertisement end up clicking it. This metric can be used to gauge how well keywords and advertisements are performing. A click through rate between 1 to 3% is generally considered as being good. The following results have been noted in the summary reports from the media relations firm:
This demonstrates that users in 2013 to 2014 and 2014 to 2015 clicked on the advertisements, and consequently were brought to the Language Portal, more frequently. |
Program documents | Since the 2011 evaluation, the Translation Bureau has continued to raise awareness and promote the Language Portal. |
| Evidence that the upcoming redesign of the Language Portal will enable users to access resources in both official languages | See Appendix C: Summary of Evaluation findings for the Department of Canadian Heritage for analysis. | Interviews and program documentation | See Appendix C: Summary of Evaluation findings for the Department of Canadian Heritage for analysis. |
| Evidence that the development, maintenance, and promotion of the Language Portal meets the evolving subject matter and e-learning professional and technical standards | Advancements in technology have created opportunities to modernize the fields of learning and education in order to increase language awareness and enhance second language communication. Recent advances such as gamification, which is the application of elements of game playing to other areas of activity (example learning), as well as links to social media entice users to frequent the online tools and resources in order to build on their knowledge, and continue to learn. The Language Portal is equipped with such elements and plans to expand and continue to develop modern, interactive learning features for the website.
Languages evolve over time and the Language Portal Division surveys the latest developments in the industry. When new terms are brought to light, vocabulary uses arise or common spelling errors become more common place, articles are added to the writing tools. For example:
Approximately 450 new content items are made available on the Language Portal each year. In addition, over 200 new links to external content were added to the Language Portal in each of the 2013 to 2014 and 2014 to 2015 fiscal years. The Language Portal conforms to broad Government of Canada web accessibility standards that include accessibility standards for persons who are visually impaired. All of the websites of the Language Portal are expected to be accessible on mobile devices once the Language Portal is migrated to Canada.ca. The Language Portal is continuously updated to respond to the web standards of the Government of Canada. The creation of a linguistic database with the help of an internal search engine that allows users to find the writing tools in both official languages will continue to be improved and access to it will be simplified. The Language Portal Division has added a number of social media functionalities to the Language Portal to align it with the evolution of web usage through: website sharing; sharing of virtual cards on Facebook and Twitter; daily Tweets; and Rich Site Summary feeds and notices by email informing users on the monthly additions to the Language Portal. |
Literature review, interviews and program documentation | Technology is enabling advancements in education, and the Language Portal is continuously being updated to take advantage of the latest developments. New content is added to the Language Portal regularly, reflecting the evolving nature of language-related subject matter. The Language Portal adheres to current web standards for the Government of Canada and looks to make use of web-based advancements as they become available. |
To what extent have Canadians benefited from using the Language Portal and its language resources?
| Indicator | Findings | Source | Conclusion |
|---|---|---|---|
| Percentage of users, by frequency, geographic location and language | The Language Portal sees more visitors from English browsers and the majority of its Canadian users are from Ontario and Quebec. As well, 64% of the visits to the Language Portal were from outside of Canada between September 20, 2014 and September 20, 2015.
Usage data organized by geographic location and browser language for fiscal year periods prior to September 2014 was not available at the time of the evaluation. Instead, data is presented for a full calendar year, from September 20, 2014 to September 20, 2015. Visits by browser language
Visits by country
Visits by Canadian province or territory
Visits by city
Visits by referrer types (how users access the Language Portal)
The visits referred from other sites create more traffic on the French pages of the Language Portal than on the English pages. |
Program documentation | Canadians access the Language Portal more than users from any other country, although they do not represent the majority of users. International visits to the Language Portal suggest that Canada is viewed as a source of language expertise. |
| Increased number of visits to the linguistic tools | Visits to the Language Portal and its content has been increasing. As shown in Appendix B Table 9, during 2014 to 2015 the users of the Language Portal visited over 4.2 million pages which was 13.5% more than the previous year at 3.7 million page views and up from just 1 million visits in 2011.
There were over 500,000 additional visits to the language content and quizzes from 2013 to 2015. |
Program documentation | Visits to the linguistic tools increased over the evaluation period indicating greater use of the Portal. |
| Opinion and information from website users, including how they first heard of it, reason for visiting, content used, etc. | It was found through focus groups conducted as part of the evaluation that some users who work for the federal government had heard about the Language Portal from co-workers. Others indicated that during their academic training or research activities associated with linguistics and Canadian bilingualism (example, Bilingual Canadian Dictionary Project, National Terminology Council, etc.) they had become familiar with the Language Portal and its several language resources and tools. It was also found that some Francophone countries collaborating with the National Terminology Council were also using the Language Portal.
Users of the French of the Language Portal affirmed that they are familiar with the tools and resources due to their work and/or specific responsibilities vis-à-vis translation or language training. During these discussions, it was found that some of these users also refer to Le grand dictionnaire terminologique of the Gouvernement du Québec. However, these users agreed that the Language Portal offers more language services and tools in both official languages. The focus group participants noted that the high quality content available on the Language Portal website benefits users with established language-related needs, such as civil servants and linguistic professionals. Participants also noted that the Language Portal is used by students in academic institutions. While participants also indicated that materials on the Language Portal can sometimes be difficult to find due to the volume of available content, there was general agreement among the focus group participants that the Language Portal is beneficial to those seeking language tools and resources. It was also noted that different user groups derive unique benefits from the website. For example:
|
Document review, program documentation and focus groups | Opinions from users of the Language Portal indicate that it is a robust and trustworthy repository of official language tools and resources. Presentation of the website could be improved to facilitate users' navigation of the Language Portal. |
Demonstrating efficiency and economy
Can the Language Portal achieve the same outputs and results at a lower cost?
| Indicator | Findings | Source | Conclusion |
|---|---|---|---|
| Input/output ratios of comparable products and services in Canada and abroad | The evaluation was not able to accurately benchmark the Language Portal costs against a website with a mandate and content similar to the Language Portal. Private sector websites do not release their operating costs. A similar website run by the Government of Quebec has much larger input costs to accompany their more robust mandate and outputs, however, this site offers content primarily in French and does not contain an English-language user interface.
Appendix B Table 10 illustrates the Language Portal's total cost for fiscal years 2013 to 2015. Review of the financial data provided by the Language Portal demonstrates that the Language Portal expenditures during the evaluation period was slightly less than the planned expenditures of $3.2 million dollars annually. In 2013 to 2014, due to the initial costs associated with the revamping of the renewed initiative, more resources were dedicated directly to the Language Portal than planned. In 2014 to 2015, some Information Technology employees were transferred to another branch within the Department which reduced salary expenditures. It was found that a number of private, web-based enterprises offer similar tools and resources to those of the Language Portal. Even though most of these services are provided without cost to users, they generate revenue from advertisements on their site. In comparison, the Language Portal is not operated as a means to generate profit, but instead to help Canadians communicate in both official languages and highlight Canadian expertise in the area of official languages with visitors. In fact, the Language Portal is not allowed to use advertisements as a revenue stream, as it would contravene the Communications Policy of the Government of Canada which states that The Office québécois de la langue française provides a number of language resources and tools, such as Le grand dictionnaire terminologique and la Banque de dépannage linguistique that are similar to the French language tools and resources on the Language Portal. This website is run by the Quebec government with a goal of assisting the population in their knowledge of the French language by offering terminological and linguistic support. This website has a much broader mandate compared to the Language Portal and its annual expenses for the 2013 to 2014 and 2014 to 2015 fiscal years were over $23 million. Given the aggregated nature of these figures, it was not possible to isolate the specific costs of comparable services, such as Le grand dictionnaire terminologique and la Banque de dépannage linguistique. |
Program documentation and document review | Without an appropriate program to benchmark the Language Portal against, the evaluation was not able to accurately compare input and output ratio costs with a similar language website. |
| Perception of efficiency and effectiveness of the Language Portal as a means to increase capacity within the language sector and knowledge of the official languages (example, added value vs. trade-offs) | Senior program management view the Language Portal as a "public good" (that is, providing free official language tools and resources) which makes it difficult to find comparable services to benchmark against, as many online language resources are profit driven and do not ensure the same protection of user information as Government of Canada websites. A comparison with the language site operated by the Government of Quebec (see below) provides some evidence that the Language Portal is being operated efficiently. The leveraging of the TERMIUM Plus® website, which was in place prior to the Language Portal being made available online to Canadians in 2009, as a foundation for expanding to the current incarnation of the Language Portal also provides an indication of efficient use of resources.
There was general consensus among the focus group participants that the Language Portal is effective in contributing to the use of official languages in Canada. |
Program documentation, focus groups and interviews | The Language Portal is perceived as being an efficient and effective tool for delivering language resources to users. |
| Ratio of administrative costs to a comparable, similar program/Portal | The evaluation team examined the administrative costs of running the Language Portal compared to a similar government online language website. As noted earlier in the report, the Office québécois de la langue française has different programs and services compared to the Language Portal, and therefore cannot be benchmarked against the Language Portal to make a direct comparison of their input costs.
Appendix B Tables 11 and 12 outlines the administrative costs and associated ratios of running the two aforementioned websites. As shown, in 2014 to 2015 both programs had similar administrative ratios at roughly 40% of their total expenditures. The Office québécois de la langue française however administers a number of other initiatives which go over and above the types of services that are provided by the Language Portal. In view of these differences, the comparative ratios provided here may not be entirely accurate. |
Document review | The Language Portal had comparable administrative ratios to a similar program in 2014 to 2015, at 40% of total budget. While there are notable differences between the two programs, the similarity in administrative cost ratios provides some confidence that the Language Portal is being delivered efficiently. |
Are there alternative arrangements for reaching the same results more cost-effectively?
| Indicator | Finding | Source | Conclusion |
|---|---|---|---|
| Evidence of viable alternative delivery methods | Program management highlighted the uniqueness of the Language Portal and its interest in providing language resources to Canadians at no cost to users. They feel the private sector would not share the same intentions as the federal government in delivering free bilingual language content with the same expected outcomes as the Language Portal. As noted in the relevance section, the 2015 Mandate Letter for the Minister of Canadian Heritage included a priority for that Minister to establish a free, online service for learning and retaining English and French as second languages. Discussions will continue in 2016 as to whether the Language Portal platform could be used to achieve this priority, thereby allowing the Department of Canadian Heritage to capitalize on the existing platform and allowing for economy and efficiency in delivering this priority.
Given the broad range of professionally-developed content offered by the Language Portal (example terminology bank, articles, quizzes, style guides and other writing tools, etc.), the evaluation was not able to identify viable alternative delivery methods for the Language Portal. Other language websites reviewed by the evaluation team did not provide for meaningful comparison as many of the websites contain advertisements, do not specify the country of origin, do not certify that the contents were prepared by accredited language professionals, or do not provide for equal representation of both of Canada's official languages. A 2013 study commissioned by the Translation Bureau was conducted to compare other language-related websites to the Language Portal and propose improvements to the Language Portal. Five websites were chosen: linternaute.com, reverso.net, Google, Wiktionnaire and WordReference. Among the websites, different options available to users were identified such as: linking capabilities to Facebook, a collaborative dictionary enabling users to create their own entries where other users can vote on the proposed terminologies, members can create their own online profiles, a toolbar that is integrated into the users internet browser, voice searches, language recognition, audio word pronunciation, choices of multiple translations, pictures, paronyms, and the ability for users to save their searches. These options were presented to program management and an action plan was subsequently developed to prioritize future improvement to the Language Portal. |
Document Review and interviews | The evaluation team did not find evidence of alternative delivery methods for the Language Portal as its delivery and mandate is such that it is best delivered by the federal government. Ongoing discussions with the Department of Canadian Heritage may allow for inter-departmental economy and efficiency in delivering on official language priorities. Opportunities exist to capitalize on features used by other similar websites run by the private sector. |
| Potential improvements to program delivery to increase efficiency | Program management noted that future efficiencies would be gained by building cost-sharing partnerships with the provinces and territories through their official language minority communities to produce more content for the Language Portal without increasing input costs. They also noted that because of the requirements of Canada.ca, there will be a steady stream of content improvement and updating. This continued process enables the Program to avoid the dramatic increase in input costs that occur when large changes to the website are required. | Interviews | The Language Portal Division is aware of opportunities to improve program efficiency, and has incorporated the efficient use of resources into its future work plans. |
Conclusion: Performance (outcomes and efficiency and economy)
The evaluation found that the Language Portal is achieving its immediate outcome of providing Canadians with access to quality language resources in both official languages and making available free linguistic resources and tools to Canadians. Improvements to the website could be made to make it more user friendly.
Limited evaluation data is available to gauge to what extent the Language Portal is meeting its intermediate outcome of Canadians benefiting from using the Language Portal of Canada and its language resources. It was noted from website users that information provided through the Language Portal is considered to be high quality. The vastness of the information can be overwhelming however, and some users noted that other sites enable them to find information more efficiently.
The Language Portal is a unique offering ("public good") of the federal government and a comparison with profit driven private sector language services would not provide for meaningful analysis. The Language Portal has the objectives of helping Canadians communicate in both official languages, disseminate and promote language resources developed in Canada and share and highlight Canadian expertise in the area of language. Other language websites reviewed by the evaluation team did not provide for meaningful comparison as many of these websites are for profit and contain advertisements, do not specify the country of origin, and do not certify that the contents were prepared by accredited language professionals, or do not provide for equal representation of both of Canada's official languages. Evidence suggests that the Language Portal has allocated its resources efficiently to provide a range of language tools and resources that are used by Canadians and users abroad.
The migration of the Language Portal to Canada.ca may further enhance its accessibility, reach, and content without increasing input costs.
Language Portal Division workplan for the two remaining years of the Roadmap for Canada's Official Languages (2013 to 2018) funding
As this evaluation was conducted halfway through the 2013 to 2018 Roadmap funding, the Language Portal Division has prepared for work to be undertaken for fiscal years 2016 to 2017 and 2017 to 2018. The Language Portal Division is preparing for a redesign of the website and other activities to continue to improve the performance of the Language Portal.
Specifically, future planned work and improvements include:
- conducting an ongoing user survey on the Language Portal to help measure intermediate and ultimate outcomes
- monitoring performance and setting appropriate targets (using Specific, Measurable, Attainable, Relevant and Time-bound indicators)
- reviewing the way content is presented on the website so that it meets the needs of a broad range of users beyond federal employees (students, new Canadians)
- further increasing awareness of the Language Portal
Recommendation
No substantive issues were uncovered over the course of the evaluation which would require formal recommendations. The evaluation team would encourage the Language Portal Division to continue the implementation of its current work plan so as to keep delivering the Language Portal for the benefit of Canadians.
Appendix A: About the evaluation
Authority
This evaluation was included in the 2013 to 2018 Risk-Based Audit and Evaluation Plan. Initially, the evaluation was to be conducted in 2016 to 2017 to provide input to the Horizontal Evaluation of the Roadmap for Canada's Official Languages 2013 to 2018: Education, Immigration and Communities led by the Department of Canadian Heritage. However, at the request of the Department of Canadian Heritage, the planning and conduct of this evaluation was expedited to allow for a Final Evaluation Report to be submitted to the Department of Canadian Heritage before June 30, 2016.
Evaluation Objectives
The evaluation examined the Language Portal of Canada, delivered by PSPC Translation Bureau. This evaluation had two objectives:
- to determine the relevance of the Program: the continued need for the Program, its alignment with governmental priorities and departmental outcomes and its alignment with federal roles and responsibilities
- to determine the performance of the Program: the achievement of its expected outcomes (immediate and intermediate) and a demonstration of efficiency and economy. The assessment of the Language Portal's outcomes in the evaluation was limited to the short-term outcomes in order to meet the June 30, 2016 reporting deadline set by the Department of Canadian Heritage
Approach to the Evaluation of Language Portal
The Office of Audit and Evaluation Directorate completed a risk-based calibration assessment to determine the best approach for the conduct of this evaluation to appropriately address the core issues identified by the Treasury Board for evaluations (continued Need; Alignment; Effectiveness; Efficiency and Economy). Based on this assessment, a Terms of Reference for the evaluation was developed. This Terms of Reference provided the requisite technical framework for the conduct of this evaluation.
The level of effort allocated to the conduct of the evaluation reflected the above factors. An evaluation matrix was developed using the Treasury Board's five core issues to be addressed in evaluations. The lines of evidence outlined below were implemented to assess the evaluation issues and questions.
- Key Informant Interviews (n=6):
- Interviews were conducted with program management and staff. Questions focused on the performance and effectiveness of the Language Portal.
- Focus Group (n=12):
- The evaluation team conducted three virtual focus group discussions with a sample of Language Portal users and ambassadors.
- Document Review:
- The preliminary document review provided information on the Language Portal and its context to assist in the planning of the evaluation. A more comprehensive program document review was conducted to collect information to answer the evaluation questions. The program document review focused on several types of internal and external documents, reports and other supporting documents. Some of these included: Treasury Board submission and related departmental documents; program administration, monitoring and reporting documents; program-led studies; financial records; as well as annual Reports on Plan and Priorities and Departmental Performance Reports.
- Literature Review:
- This comprised of a web-based literature search to facilitate contextualizing the Program, as well as identifying alternative delivery models. The literature review assisted in structuring the key informant interviews, focus group discussions, and costing analysis.
- Financial Review:
- Financial data related to the program's budgets, revenues, and expenditures were examined to assess the efficiency and cost-effectiveness of the Program.
- Performance Data Review:
- The Program's performance data was examined to determine to what extent the Program is achieving its intended immediate and intermediate outcomes.
Limitations of the Evaluation
Roadmap funding for the Language Portal is from April 2013 to March 2018, and the evaluation results are required by June 2016. This means that the evaluation was not able to assess the five year (2013 to 2018) performance of the Language Portal under the Roadmap funding.
Furthermore, there are inherent challenges associated with measuring the outcomes of initiatives focused on increasing knowledge and use of languages, such as the Language Portal, due to the complex nature of language acquisition, development and use, and also external factors that contribute to these processes.
The logic model for the Language Portal identifies the following as its immediate outcome: "Canadians have access to quality language resources in both official languages". Given the subjective nature of establishing the quality of a product or service, the evaluation was conducted from the perspective that the Language Portal Division, and professionals employed within the Translation Bureau, possess the requisite skills and abilities to produce quality language resources. Consequently, the evaluation did not conduct direct assessments of the quality of the language products available from the Language Portal. Instead, the quality of resources was assessed indirectly through Language Portal utilization data and focus group sessions.
Reporting
Based on the lines of evidence, the evaluation team prepared a Director's Draft Report for Validation presenting the findings and conclusions. This document was reviewed by the Quality Assessment and Practice Improvement Directorate within the Office of Audit and Evaluation and by the Chief Audit and Evaluation Executive. Subsequently, it was sent to senior program management for review and comment. Validation of facts and any other comments on the Director's Draft Report for Validation were assessed and, where appropriate, incorporated in the Chief Audit and Evaluation Executive's Draft Report for Validation. This document was sent to the Office of Primary Interest for final acceptance. A management response to the Chief Audit and Evaluation Executive's Draft Report for Validation was requested. The Final Report for Validation, including the management response, was tabled at the Public Services and Procurement Canada Audit and Evaluation Committee for the Deputy Minister's approval in June 2016. The Final Report will be sent to the Department of Canadian Heritage, submitted to the Treasury Board Secretariat, and posted on the PSPC website.
Project team
The evaluation was conducted by an employee of the Office of Audit and Evaluation and a consultant, overseen by the Director of Evaluation and conducted under the overall direction of the Chief Audit and Evaluation Executive. The evaluation was reviewed by the Quality Assessment and Practice Improvement Directorate of the Office of Audit and Evaluation.
Appendix B: Tables
| Indicators | 2013 to 2014 | 2014 to 2015 | Percentage change |
|---|---|---|---|
| Page views | 3.75 million | 4.2 million | 12% |
| New content available added | 481 | 468 | -2.70% |
| Total content available | 2,956 | 3,424 | 15.83% |
| New headlines, language articles and tips of the week | 287 | 286 | -0.34% |
| Total headlines, language articles and tips of the week | 2,129 | 2,415 | 13.43% |
| New quizzes | 126 | 128 | 1.59% |
| Total quizzes | 593 | 721 | 21.59% |
| New contributor articles | 68 | 54 | -20.59% |
| Total contributor articles | 234 | 288 | 23.08% |
| Writing tools | 16 | 16 | 0% |
| New hyperlinks | 205 | 203 | -0.98% |
| Total hyperlinks | 1,936 | 2,139 | 10.49% |
| Ambassadors | 0 (The Ambassador Program was launched in 2014 to 2015) |
60 | 100% |
| Tools and resources (French + English) | Number of page views 2013 to 2014 | Percentage of page views 2013 to 2014 | Number of page views 2014 to 2015 | Percentage of page views 2014 to 2015 |
|---|---|---|---|---|
| Home page and promotional pages | 509,913 | 13% | 420,943 | 10% |
| Language content (page with a link to TERMIUM Plus® and the Writing tools; Le français sans secrets/Gateway to English; Headlines; Tip of the Week; external language resources; Contributors' articles) | 2,458,783 | 66% | 3,004,243 | 72% |
| Quizzes | 777,526 | 21% | 759,685 | 18% |
| Total | 3,746,222 | 100% | 4,184,871 | 100% |
| Items | Allocated | Actual | Variance | |||
|---|---|---|---|---|---|---|
| 2013 to 2014 | 2014 to 2015 | 2013 to 2014 | 2014 to 2015 | 2013 to 2014 | 2014 to 2015 | |
| Salary | $1,782,237 | $1,837,821 | $2,180,312 | $1,572,254 | $-398,075 | $265,567 |
| Operation and maintenance | $1,317,763 | $1,362,179 | $881,753 | $1,522,118 | $436,010 | $-159,939 |
| Total | $3,100,000 | $3,200,000 | $3,062,065 | $3,094,372 | $37,935 | $105,628 |
Table 11: Sample input/output ratio of the Language Portal by costs related to operations and maintenance, total expenses and the administrative ratio for 2013 to 2014 and 2014 to 2015
| Fiscal year | 2013 to 2014 | 2014 to 2015 |
|---|---|---|
| Operation and maintenance | $881,753 | $1,522,118 |
| Total expenses | $3,062,065 | $3,094,372 |
| Administrative ratio | 28.80% | 49.19% |
Table 12: Sample input/output ratio of the Office québécois de la langue française by costs related to operations and maintenance, total expenses and the administrative ratio for 2013 to 2014 and 2014 to 2015
| Fiscal year | 2013 to 2014 | 2014 to 2015 |
|---|---|---|
| Operation and maintenance | $10,104,600 | $9,853,900 |
| Total expenses | $23,088,700 | $23,179,400 |
| Administrative ratio | 43.76% | 42.51% |
Appendix C: Summary of evaluation findings for the Department of Canadian Heritage
Summary of evaluation findings for the Department of Canadian Heritage
Horizontal evaluation of the Roadmap for Canada's Official Languages 2013 to 2018
In support of its Horizontal Evaluation of the Roadmap for Canada's Official Languages 2013 to 2018, the Department of Canadian Heritage has requested that evaluations conducted by organizations receiving funding through the Roadmap address a set of common issues. This will allow Canadian Heritage to aggregate the analyses of individual evaluations products into its Horizontal Evaluation which is focused on demonstrating the relevance and performance of the Roadmap.
The Horizontal Evaluation will include an analysis of issues common to the Roadmap initiatives, namely the:
- roles and responsibilities of stakeholders
- scope of the initiative and its impact on various target audiences
- validity and reliability of performance indicators in capturing tangible results
To facilitate the Department of Canadian Heritage's task, Public Services and Procurement Canada's Office of Audit and Evaluation has prepared a summary of findings from its current Evaluation of the Language Portal of Canada aligned to the common issues identified by Canadian Heritage.
1. Roles and responsibilities of stakeholders
The 2013 to 2018 Roadmap addresses three areas that were identified as priority areas of action as part of the 2012 Consultations on Official Languages conducted by the Department of Canadian Heritage. The three areas identified were: education, immigration and support for communities. The "Education" pillar aims to support Canadians having opportunities for learning and training in their first official language, for learning the country's other official language, and for gaining access to technological tools, and that they benefit from the numerous social, economic, cultural and identity benefits resulting from these. As such, it is expected that Roadmap initiatives are linked to objectives under one of the Roadmap pillars (that is education, immigration and/or community), and that the roles and responsibilities within federal organizations contributing to the Roadmap are conducive to delivering these results.
PSPC supports the Education pillar objective of learning both official languages through the Language Portal of Canada. PSPC's Translation Bureau, which is responsible for the creation and management of the Language Portal, supports this pillar by providing Canadians, as well as private sector, governmental and non-governmental organizations, with online access to a significant number of language tools and resources intended to help them study, work and communicate more effectively in Canada's official languages. It also allows internet users from across Canada (and around the world) to access language resources developed in Canada, such as articles and quizzes to help them improve their language skills.
Though an analysis of the social, economic, cultural and identity benefits resulting from the Language Portal was beyond the scope of the evaluation, the Language Portal's role is seen as supporting "conditions in which Canadians can live and thrive in both official languages", which is relevant to the strategic outcome of the Roadmap.
The Language Portal sees millions of visitors annually - from 2013 to 2015, Language Portal web pages were viewed close to 8 million times, as compared the 2 million times its pages were viewed in the 2010 and 2011 calendar years. This rate of utilization suggests that the Language Portal of Canada has increased its effectiveness at attracting visitors to the available language content. As a result, the Language Portal's effectiveness in its role of helping Canadians communicate in both official languages and promoting Canadian language expertise can also be seen to have increased. In addition, the Language Portal's role in fostering partnerships with private sector stakeholders, different levels of government and within the federal government have contributed to increasing the amount of content available to all Canadians, including those in minority language communities nationwide.
Further, future plans for the Language Portal include an increasing role for the Language Portal in enabling collaborations with the provinces and territories, as well as with official language minority communities, which will continue to enrich the content available on the website.
Given these considerations, the evaluation found that the design and delivery of the Language Portal of Canada contributes to the achievement of the objectives of the education pillar of the Roadmap, by way of its role in making official language tools and resources available online to all Canadians for free.
2. Scope of the initiative and its impact on various target audiences
The foundation for the design and delivery of the Language Portal emerged from TERMIUM Plus®, the Government of Canada's online terminology and linguistic data bank for the federal public sector until the Language Portal was made publicly available to Canadians in 2009. The existence of TERMIUM Plus® allowed for the Translation Bureau to implement the Language Portal with a less significant level of effort than would have been required to initiate and implement a fresh website.
Building on these early efficiencies, the renewal of the Language Portal's funding has allowed the Language Portal Division to increase the amount of content made available on the website.
There were 3,424 language resources available on the Language Portal at the end of 2014 to 2015, representing an increase of 38% from the 2,475 available at the end of 2012 to 2013:
- 573 headlines, language articles and tips of the week were added in that period, representing an increase of 31%
- 254 languages quizzes were added in that period, representing an increase of 54%
- 122 contributor articles were added in that period, representing an increase of 74%
- 408 hyperlinks were added in that period, representing an increase of 24%
The evaluation also found that, in recognition of the fact that languages evolve over time, the Language Portal Division surveys the latest developments in the industry so that when new uses arise or common spelling errors become more common place, articles are added to the writing tools. For example:
- over the last five years, the Language Portal Division has updated the entire content in the "Clefs du français pratique" and "Writing Tips" as words or terms formerly considered to be mistakes have entered into common vocabulary, and dictionaries
- articles were added on new French (France) spellings when interest emerged on these changes in francophone Canada
- new articles were added to the Language Portal on the latest French (France) grammar when it started being taught in schools
- content is rewritten when it becomes apparent that usage has evolved
Accredited language professionals working for the Language Portal Division are responsible for creating 90% of the tools and resources available on the Language Portal. This includes:
- the creation of a linguistic database "Le français sans secrets | Gateway to English" permitting simultaneous searches in over 20 000 language websites (16 writing tools, hundreds of games and articles, recommendations and linguistic reminders from the Translation Bureau)
- the creation of linguistic content (headlines, linguistic articles, articles written by the collaborators and games – excluding external language resources)
Content for the Language Portal has also been developed by 288 partners/collaborators and ambassadors representing the remaining 10% of the total content available directly from the website, which serves not only to increase the amount of content available on the website, but also the scope of its impact on the larger community of language professionals and organizations. The collaborators have a mandate to produce articles that reflect their reality, their activities, their accomplishments and their field of work. Some of the organizations affiliated with the collaborators include teachers associations, writers and reviewers' associations, universities and provincial governments.
In 2011, the Language Portal was evaluated as part of the Horizontal Evaluation of the Roadmap for Canada's Linguistic Duality 2008 to 2013 - Acting for the Future. The evaluation recommended that the Translation Bureau "…consider promotional or other activities to increase the visibility of the Language Portal of Canada and awareness of its language tools and resources by all Canadians"
. In the management response, Translation Bureau committed to developing a Strategic Promotional Plan, by:
- launching a search engine based advertising campaign
- developing and implementing an outreach program in collaboration with partners, bloggers and possible ambassadors
- implementing promotional collateral activities integrating social media
- participating in various events targeting the Language Portal's audiences
The follow-up on the management action plan for the 2011 evaluation indicated that all of the action items were fully implemented, including actions which can reasonably be seen as having increased the scope of the Language Portal, such as:
- creating web links to guide users through cross-marketing to the ambassadors program
- adding links on the Language Portal site where users could send virtual cards to people on social media
- giving a presentation at the annual conference of the "Association Canadienne des Journaux des Débats"
- creating a poster for the "Rendez-vous de la Francophonie 2014"
Since 2011, the Language Portal Division has engaged a media relations firm to manage its annual search engine based advertising campaign. In describing the process, the Language Portal Division indicated that they identified thousands of keywords and terms that a person might enter in a search engine when looking for a website offering content similar to the Language Portal's. During the advertising campaigns, when users entered one of those keywords or terms into a search engine, an advertisement about the Language Portal would be presented and users would be able to click on it to be directed to the Language Portal website.
The number of clicks is monitored and transformed into a metric known as a "click through rate" which shows how often people who see an advertisement end up clicking it. This metric can be used to gauge how well keywords and advertisements are performing. A click through rate between 1 to 3% is generally considered as being good. In 2011, the advertising campaign yielded an average click through rate of 3.85%; in both 2013 to 2014 and 2014 to 2015, the average click through rate was 6.93%. This demonstrates that users in 2013 to 2014 and 2014 to 2015 clicked on the advertisements, and consequently were brought to the Language Portal more frequently.
Advancements in technology have created opportunities to modernize the fields of learning and education in order to increase language awareness and enhance second language communication. Recent advances such as gamification, which is the application of elements of game playing to other areas of activity (example learning), as well as links to social media entice users to frequent the online tools and resources in order to build on their knowledge, and continue to learn. The Language Portal is equipped with such elements and plans to expand and continue to develop modern, interactive learning features for the website.
The Language Portal conforms to broad Government of Canada web accessibility standards that include accessibility standards for persons who are visually impaired, and is continuously updated to respond to the web standards of the Government of Canada. In addition, all of the pages of the Language Portal are expected to be accessible on mobile devices once the Language Portal is migrated to Canada.ca. Further, the Language Portal Division has added a number of social media functionalities to the Language Portal to align it with the evolution of web usage through: website sharing; sharing of virtual cards on Facebook and Twitter; daily Tweets; and Rich Site Summary feeds and notices by email informing users on the monthly additions to the Language Portal.
Impacts on the Language Portal's target audience were measured through an analysis of the number of visits to the website, as well as through qualitative data generated from focus group discussions with Language Portal users and ambassadors. Overall, the evaluation determined that the website was reaching its targeted audiences, and providing Canadians with access to a vast collection of official language tools and resources. Language Portal users and ambassadors who participated in the focus group discussions indicated that the Language Portal was used in several ways by various stakeholders; students and language learners were cited as benefiting from the tools and resources available on the website, and public servants rely on TERMIUM Plus® for their terminological needs. Overall, the Language Portal immediate outcome of Canadians having access to quality language resources in both official languages is being met.
More robust data on intermediate outcome achievement was not available at the time of the evaluation. However, the amount of content available on the Language Portal provides for an impressive scope for this initiative. Further, increases in utilization rates of the Language Portal suggest that the content of the site is reaching a significant number of users, though its impact on target audiences could not be determined. The Program is developing an ongoing user survey which will provide more robust performance data related to this indicator when the Language Portal is migrated to the Canada.ca website in 2016.
3. The validity and reliability of performance indicators in capturing tangible results
The 2011 evaluation, and the current evaluation examined the performance of the Language Portal against the Program's detailed performance indicators and additional performance measures. The examination of the results from the Language Portal's performance measurement framework was used to inform the assessment of performance in both evaluations, and these quantitative indicators were deemed to be a reliable source of performance data. The evaluations expanded on the analysis in order to depict a more fulsome narrative of the Language Portal's reach and benefits. Through a literature review and other qualitative data sources, a holistic review of the performance of the website was possible.
The Language Portal Division has been collecting performance measurement data since the creation of their performance measurement strategy (as recommended by the previous evaluation of the Language Portal in 2011). There are gaps in measuring longer term outcomes, such as the intermediate and ultimate outcomes, but this is slated to be addressed in an ongoing user satisfaction survey to be launched by the end of 2016. The performance indicators used to measure the immediate outcomes are considered to be valid and reliable. The evaluation team was unable to identify a comparable program to benchmark the Language Portal against.
- Date modified: