Evaluation of the CFGB Review of Military Grievances Program

July 2010

Project Number: 570-2809
Sigma WBSE: C.000565.002
Government Consulting Services
Program Evaluation and Performance Measurement Services

List of Acronyms

Board = Canadian Forces Grievance Board
CDS = Chief of Defence Staff
CF = Canadian Forces
CFGA = Canadian Forces Grievance Authority
CFGB = Canadian Forces Grievance Board
CO = Commanding Officer
DND = Department of National Defence
F&Rs = Findings and Recommendations
FA = Final Authority
FTE = Full Time Equivalent
GCS = Government Consulting Services
IA = Initial Authority
MND = Minister of National Defence
MRRS = Management, Resources and Results Structure
NDA = National Defence Act
PAA = Program Activity Architecture
PWGSC = Public Works and Government Services Canada
QR&O = Queen’s Regulations and Orders for the Canadian Forces
RMAF = Results-Based Management and Accountability Framework
TBS = Treasury Board Secretariat

Executive Summary

The purpose of this report is to presents the findings of the evaluation of the Canadian Forces Grievance Board (CFGB) Review of Military Grievances Program. As this evaluation is summative, it examines the relevance and performance, including the effectiveness, efficiency and economy, of the Program during the five year period from January 2005 to December 2009. The research for this evaluation was conducted between February and June 2010.

Description of the Program

In the early nineties, a number of studies and working groups were initiated to help identify and propose solutions for a number of issues that existed with the military’s complaint resolution methods. As a result, the CFGB was created in accordance with amendments made to the National Defence Act (NDA) in 1998. The Board received its regulatory authority and officially began operations on June 15, 2000.

The CFGB has only one program which is the Review of Military Grievances Program. Its objective is that the findings and recommendations (F&Rs) of the CFGB be implemented and lead to improvements in the conditions of service for members of the Canadian Forces (CF). This is achieved through the review of CF grievances which have been referred by the CDS in a manner which is expedient, fair, transparent and according to the law, and the communication of case summaries, lessons learned, trends and systemic issues.

Methodology

The evaluation methodology integrated the use of multiple lines of evidence and complementary quantitative and qualitative research methods as a means to ensure the reliability of results being reported, and the validity of information and data collected. The research methods included:

document and literature review;
stakeholder interviews;
a focus group;
review of performance data, including both CF & CFGB surveys; and
review of financial data.

The evaluation focused on program relevance, effectiveness, efficiency and economy.

Relevance

The Review of Military Grievances Program at the CFGB continues to be relevant. The need for an external review of military grievances is legislated and the CDS , who receives the CFGB’s recommendations, perceives a continued need. However, as noted in the Frecker report, there is considerable duplication in the work of the Canadian Forces Grievance Authority (CFGA) and the CFGB, in relation to analysis of the grievances.

The objectives of the CFGB align with the government priority Rebuilding the Canadian Forces. The CFGB activities align with federal roles and responsibilities, since the federal government mandate clearly includes the creation and maintenance of an armed force. The CFGB supports this and is therefore aligned with federal roles and responsibilities. However, awareness activities currently being conducted by the CFGB and follow-up on recommendations on systemic issues are within the legislative intent only to the extent that they support the provision of findings and recommendations.

Recommendation: The CFGB should examine its activities to ensure they align with its mandate.

Performance (Effectiveness)

The CDS has an increased awareness and understanding of systemic issues arising from grievances reviewed by the CFGB. He also agrees that the CFGB F&Rs assist him in rendering decisions. Interviewees noted that the analysis in the F&Rs was comprehensive and of high quality and that they present the reasoning behind the recommendations.

Recommendation: The CFGB should evaluate the effectiveness of the various outreach activities

There is no clear evidence that the CFGB’s work has lead to improvements to regulations, policies, guidelines and the grievance process. Although, perceptions are that it has. Similarly, there is no clear evidence that confidence levels in the grievance process are increasing, nor is it clear whether the CFGB is having an impact on this confidence. Nonetheless, grievors seem satisfied with the fairness, transparency and equity of the process, although less satisfied with timeliness. It is not possible to know the extent to which CFGB F&Rs have been implemented, as the CFGB is not made aware of actions taken in response to F&Rs.

Performance (Efficiency & Economy)

The average cost per F&R, when considering the CFGB budget overall is $50,141, which seems significant. However, this may be the cost required to maintain an independent organization. There are some areas where the CFGB seems to be working efficiently and has made improvements. The CFGB has been regularly examining its processes in an effort to make them as efficient as possible. For instance, improvements made to the process have meant that over the past five years the time to process a grievance has consistently dropped, going from 417 days in 2005 down to 199 days in 2009.

However, in spite of the improvements already made to the process, stakeholders still suggest there are better ways to achieve the outcomes, particularly with regards to removing some of the duplication of effort between the CFGB and the CFGA.

Recommendation: The CFGB should continue to work with the CFGA to streamline the process and remove duplication.

CFGB spending and FTEs seems to be overly weighted towards internal services. This may be as a result of how CFGB attributes costs within the organization, but further analysis would be required to determine this. Anecdotally, it has also been reported that some grievance officers are not reporting their time, which makes decision making regarding spending less precise.

Recommendation: The CFGB should examine the allocation of spending and FTEs on internal services versus the review of grievances. This includes ensuring that the CFGB has the best possible information available regarding the cost of operations.

1. Introduction

This report presents the findings of the evaluation of the CFGB Review of Military Grievances Program. This evaluation was undertaken by Government Consulting Services (GCS) for the Canadian Forces Grievance Board (CFGB) between February and June 2010. As this evaluation is summative, it examines the relevance and performance, including the effectiveness, efficiency and economy, of the Program during the five year period from January 2005 to December 2009.

The evaluation followed the scope and methodology set out in an evaluation plan which was developed by GCS between November 2009 and January 2010. This plan was aligned with the Treasury Board of Canada Secretariat (TBS) Policy on Evaluation which came into effect April 2009.

The evaluation report is organized as follows:

Section 1 presents a profile of the CFGB and the CFGB Review of Military Grievances Program;
Section 2 presents the methodology for the evaluation;
Section 3 presents conclusions and supporting findings by evaluation issue and question; and
Section 4 presents a summary and recommendations.

1.1 Program Background

In the early nineties, a number of studies and working groups, internal to the Canadian Forces (CF), were initiated to help identify and propose solutions for a number of issues that existed with the military’s complaint resolution methods. A series of recommendations ensued, including those put forth by the Doshen Report, which were aimed at modernizing the system as a whole. In 1996, the Armed Forces Council ordered a streamlined grievance system be developed and in 1997 the Minister of National Defence (MND) submitted his report to the Prime Minister on the leadership and management of the CF. This particular report included two recommendations relating to the grievance system: the removal of the Minister as final arbiter and the creation of an independent review board.

As a result of these and other studies and inquiries, including the Somalia Commission of Inquiry, the CFGB was created in accordance with amendments made to the National Defence Act (NDA) in 1998. The Board received its regulatory authority and officially began operations on June 15, 2000.

1.2 Overview of the Canadian Forces Grievance System^{Footnote 1}

The streamlined Canadian Forces Grievance System which resulted from the 1998 revisions includes two decision-making points: the first decision level, (Initial authority (IA)), and second, (Final authority (FA)). According to existing orders and regulations, a CF member has six months to submit their grievance through their Commanding Officer (CO) from the time of the decision, act or omission that is the object of their grievance. If the CO cannot act as the IA because they do not possess the required authority to remedy the grievance, or because they may be personally involved in the matter being grieved, the grievance must be forwarded to the proper IA within ten days.

The IA has 60 days to consider and decide on the grievance unless a request is made to the grievor for an extension and the extension is granted. The grievor may request to have their grievance forwarded to the FA if the IA has not been able to consider the grievance within the allowed timeframe. If not satisfied with the decision of the IA, the grievor may request that the grievance be forwarded to the FA for consideration and determination. In such a case, the grievor has 90 days from receiving the decision of the IA to request that the grievance be submitted to the FA.

The Canadian Forces Grievance Authority (CFGA) reviews all grievances received at the FA level to determine whether a grievance is among those which must be referred to the CFGB. As stipulated in the NDA and article 7.13 of the Queen’s Regulations and Orders for the Canadian Forces (QR&O), the Board’s mandate is to review all military grievances referred to it by the Chief of the Defence Staff (CDS). QR&O 7.12 sets out the types of grievances that may be referred to the Board. Specifically:

(1) The Chief of the Defence Staff shall refer to the Grievance Board any grievance relating to the following matters:

Administrative action resulting in the forfeiture of, or deductions from, pay and allowances, reversion to a lower rank or release from the Canadian Forces;
Application or interpretation of Canadian Forces policies relating to expression of personal opinions, political activities and candidature for office, civil employment, conflict of interest and post-employment compliance measures, harassment or racist conduct;
Pay, allowances and other financial benefits; and
Entitlement to medical care or dental treatment.

(2) of the Chief of the Defence Staff in respect of a particular officer or non-commissioned member to the Grievance Board for its Findings and Recommendations.

The Chief of the Defence Staff shall refer every grievance concerning a decision or an act

Section 29.12 of the NDA stipulates that the CDS may also refer any other grievance to the Board.

These particular files on average account for approximately 40% of all the grievances received at the FA level. To review these grievances the Board has quasi-judicial powers and may summon witnesses and compel them to give oral or written evidence. Although hearings are normally held in private, the Chair may deem a hearing public, if it would benefit the participants and serve the public interest.

Following its review, the Board submits its Findings and Recommendations (F&Rs) to the CDS , and simultaneously forwards a copy to the grievor. Prior to their being transferred to the CDS , however, the CFGA reviews the file and determines the CF response to the CFGB F&R. It is nonetheless the CDS who is the final authority on all military grievances mandatorily referred to the Board. The CDS is not bound by the Board’s F&Rs, but must provide reasons, in writing, in any case where he or she does not action an F&R.

Files that are not included in the 40% referred to the CFGB stay within the CFGA and are, for the most part, adjudicated by its Director General (DGCFGA) who has the delegated authority to act as the FA on behalf of the CDS .

Decisions of the FA are final and binding but for judicial review by the Federal Court of Canada (FCC).

1.3 Profile of the CFGB Review of Military Grievances Program

As per the logic model found in Appendix A, the overall objective of the CFGB Review of Military Grievances Program is that F&Rs of the CFGB be implemented and lead to improvements in the conditions of service for members of the Canadian Forces.

This objective is achieved through two activities:

The review of Canadian Forces grievances which have been referred by the CDS in a manner which is expedient, fair, transparent and according to the law, and
The communication of case summaries, lessons learned, trends and systemic issues.

When the CDS refers a grievance to the CFGB, it is reviewed to determine whether there are precedents on the matter and whether the chain of command has correctly interpreted the applicable guidelines and regulations. The Board also ensures the process is fair and the decisions made by the CF are reasonable. The Board then prepares a report of findings and proposes recommendations to the CDS on how the grievance might best be redressed. When the Board perceives that the subject matter of a grievance is a broader, systemic issue, the Board will highlight this and make a recommendation to the CDS as to how this larger issue might be resolved. Then, so as to ensure that CF members benefit from the recommendations of the CFGB, the Board publishes case summaries, lessons learned, trends and notices of its recommendations on systemic issues.

Through remedies suggested by the Board, increased awareness of systemic issues and increased knowledge of procedures and policies, it is hoped that the application and interpretation of CF regulations, policies, and guidelines are improved. It is also expected that with greater knowledge and understanding of the regulations and guidelines, as well as with the resolution of systemic issues and grievances, stakeholders are increasingly confident that the system to consider and resolve grievances in the CF is fair.

2. Evaluation Issues and Methodology

The logic model, evaluation questions and indicators for the CFGB Review of Military Grievances Program evaluation were developed, in consultation with stakeholders, as part of the development of the evaluation framework. The evaluation was conducted according to the evaluation matrix presented in Appendix B. Below is a summary of the evaluation issues and questions.

2.1 Evaluation Issues and Questions

The questions for this evaluation have been designed to focus on the relevance and performance of the CFGB Review of Military Grievances Program as per the TBS Evaluation Policy. A summary of the research questions for the evaluation is as follows:

Relevance

Is there a continued need for an external review of military grievances?
Do the objectives of the CFGB continue to align with government priorities?
Do the CFGB's activities, outputs and outcomes align with federal roles and responsibilities?

Performance (effectiveness, efficiency and economy)

To what extent does the CDS have an increased awareness and understanding of systemic issues arising from grievances referred to the CFGB?
To what extent does the CFGB assist the CDS in rendering decisions on grievances?
To what extent has the work of the CFGB contributed to an increase in stakeholders’ awareness and understanding of the CF grievance process and related regulations, policies and guidelines affecting CF members?
As a result of the CFGB’s work, to what extent have regulations, policies, guidelines and the grievance process been improved?
What impact has the work of the CFGB had on confidence levels in the grievance process?
To what extent have the F&Rs of the CFGB been implemented and led to improvements in the conditions of service for CF members?
What factors contributed or detracted the CFGB from completing its work and reaching its objectives?
Are there more effective and efficient means of achieving the CFGB's objectives?
Is the CFGB delivering this program efficiently? How could this be improved?

2.2 Data Collection Methods

The Evaluation Matrix integrates multiple lines of evidence and complementary research methods as a means to ensure the reliability of the information and data to be collected.

Four main data collection methods were used. These lines of inquiry included quantitative and qualitative methods as follows: document review; analysis of quantitative information from existing program data; interviews and a focus group. A description of each data source is described below by line of inquiry.

2.2.1 Document Review

Policy, planning and reporting documents were reviewed and analyzed to assess the continued relevance and effectiveness of the CFGB Review of Military Grievances Program. The following types of documentation were identified for review during the evaluation:

Government priorities and policy documents;
Departmental performance reports;
Applicable legislation and regulations;
Reports and articles regarding the CFGB; and,
Communications and outreach documents.

Both hard-copy and electronic documents were reviewed during the document review. For information about the specific indicators addressed by the document review see Appendix B. The list of documents reviewed is found in Appendix C.

The document review was conducted using a customized template to extract relevant information from the documents and organize it according to indicators and evaluation questions.

2.2.2 Interviews

Interviews were conducted during the evaluation to gather opinions on various aspects of the CF grievance process, as illustrated in Appendix B. They served as an important source of information for the evaluation, providing qualitative and quantitative input on program relevance and results. A total of 19 interviews were completed during the course of the evaluation. Key informants included current and past CFGB chairs and directors, current and past decision makers within the CF and a selection of counsel for recent grievors. The distribution of interviewees was as shown in the following table.

Table 1. Distribution of Interviewees
Interview Group	Number of Interviews
External	14
– current & former decision makers	11
– counsel of grievors	3
Internal	5
– CFGB Chairs	3
– Directors	2
Total	19

Given that not all stakeholders were interviewed, the evaluation team selected a sample to ensure that appropriate interests and organizations were represented. All interviews were conducted in person. Interviewees were contacted in advance of the interview to schedule an appropriate time and were sent an interview guide. The interview guides can be found in Appendix D. The findings of the interviews were compiled and summarized by evaluation question and indicator.

2.2.3 Focus Group

A focus group was conducted to gather the perceptions of program staff as to the effectiveness, efficiency and economy of the CFGB Review of Military Grievances Program processes, as illustrated in Appendix B.

This focus group was limited to four people. Given that not all staff were invited to participate, the evaluation team selected a sample to ensure that appropriate perspectives were represented. Focus group participants were contacted in advance and were provided with a focus group guide. The focus group guide can be found in Appendix E. The findings of the focus group were compiled and summarized by evaluation question and indicator.

2.2.4 Data Analysis

Administrative, financial and performance data was gathered as specified in the evaluation matrix in Appendix B, to help assess the effectiveness and economy of the CFGB. The primary source for quantitative information was the Board’s case management database, the data from the grievors’ surveys, the Canadian Forces “Your Say” survey and financial systems. In completing the review of the administrative and performance data, GCS summarized the performance data results by evaluation question and indicator.

2.3 Limitations

The evaluation methodology was designed to provide multiple lines of evidence in support of evaluation findings. The data and information was collected to respond to the evaluation questions and issues. As in all evaluations, there are limitations and considerations that should be noted.

Limited External Analysis of the CFGB

The document review was not able to gather the perceptions of many external stakeholders as the CFGB has not been the subject of many reports or articles. Given that the Board has only been in existence ten years and acts within an internal process of the CF this is not a surprise. To account for this lack of external commentary, the external stakeholder group for interviews was sizeable.

Reporting time to Grievances

As part of their performance measurement, grievance officers are to report the time they spend on each grievance. This allows the Board not only to quantify the utilization of its staff, but also to calculate the cost to process a grievance. Unfortunately there has been a measure of delinquency among the grievance officers in the reporting of their time. Thus, the figures provided by the CFGB for this evaluation, regarding utilization and cost per grievance, contain errors and a valid calculation to account for this error has not been established. Therefore, less precise financial information has had to be used, in lieu, to assess efficiency and economy.

CF members could not be surveyed

A survey of the members of the CF is not possible given public opinion restrictions on this population. In lieu, the evaluation analyzed questions from the “Your Say” survey which is administered annually by the CF.

Change of Strategic Outcome

The logic model for this Program was revised in the months prior to the evaluation, including a change to the Board’s Strategic Outcome. The evaluation matrix upon which this report is based was in turn developed based upon the revised logic model. Part way through the evaluation, it was determined that the new Strategic Outcome required further revisions. Given that the revised Strategic Outcome may or may not have been approved before the completion of the evaluation report, it was decided that the evaluation should report instead on the previously approved Strategic Outcome. While the wording of the two Outcomes differed, their spirit was consistent, so the data collected remained relevant and it was possible to report against the previous Strategic Outcome.

Conclusion

While there are some limitations with the evaluation methodology, GCS identified mitigation strategies where possible, and designed the evaluation to use multiple lines of evidence to draw conclusions about the CFGB Review of Military Grievances Program, thus strengthening the reliability and validity of the evaluation results. Despite the limitations, the methodology meets the requirements of the policy and associated standards for evaluation.

3. Conclusions & Supporting Findings

Found in this section of the report are conclusions and supporting evidence in relation to relevance, effectiveness, efficiency and economy. Each sub-section is further broken down by evaluation question. A conclusion is stated for each of the evaluation questions followed by a description of the supporting evidence.

3.1 Relevance

The following section seeks to determine the extent to which the CFGB and the Review of Military Grievances Program address a demonstrated need, is appropriate to the federal government, and is responsive to the needs of Canadians.

3.1.1 Is there a continued need for an external review of military grievances?

Conclusion – There is a continued need, given that external review of military grievances is legislated. As well, the mandate of the CFGB is to provide findings and recommendations to the CDS , and the CDS perceives a continued need. A continued need is further supported by the fact that the CFGB has discretionary files referred to them, which suggests the CDS finds their work valuable as well. However, as noted in the May 2008 report by John Frecker of Legistec Inc., the overlapping mandates of the CFGB and the DGCFGA ... give rise to considerable duplication in relation to analysis of the grievances that are referred to the CFGB for F&Rs. ^{Footnote 2}

Supporting Evidence – The CFGB mandate is legislated in the NDA and QR&O, as described in 1.2. Therefore, to meet this obligation the CFGB is needed. The purpose of the CFGB’s F&Rs is to provide recommendations to the CDS , and he, like many of the interviewees (12 of 19), felt that gaps would exist in the CF grievance system without the CFGB. These gaps would include the loss of an external viewpoint (as noted by 8 of 19 interviewees), and the loss of the perception of credibility and independence (7 of 19 interviewees). Those who felt gaps would not exist suggested that sufficient capacity exists or could be built up within the CFGA (7 of 19 interviewees) and that the work of the CFGA is fair (5 of 19 interviewees). A quarter of CF stakeholders (2 of 8) felt that the role played by the CFGB was fulfilled elsewhere, by the CFGA.

In May 2008, John Frecker of Legistec Inc. was contracted by the CFGB to review the Board’s internal case management process. During this review he noted that The overlapping mandates of the CFGB and the DGCFGA make the formal CF grievance process quite complex and give rise to considerable duplication in relation to analysis of the grievances that are referred to the CFGB for F&Rs.^{Footnote 3} While the CFGB has no say in how the CFGA interprets its own mandate, the establishment of parallel review structures and the repetition of analysis on files, suggests unnecessary duplication.

Nonetheless, the CFGB is playing an important role with regards to fairness, as it agrees with the grievor on average 41.2% of the time, implying that it is finding issues not addressed or incorrectly addressed by the chain of command some of the time. Furthermore, by maintaining a database of F&Rs it is able to identify and communicate on average 20 recommendations on systemic issues per year to the CDS . Thus the CF is alerted to areas of confusion, which if resolved, could prevent future grievances. Finally, since 2007, the CFGB has had discretionary files referred from the CDS beyond those which they are mandated by the QR&O to review. These have represented 15.0% of the CFGB case load, and imply that the CDS finds there is value-added in the work of the CFGB.

Similarly, most interviewees (16 of 17) agreed that the CFGB activities align with the federal government values of equity, transparency and fairness. As evidence of this they cited how the CFGB provides for due process for grievors (3 of 16), disclosure of the results to the grievors (4 of 16), and access to an impartial board (3 of 16).

3.1.2 Do the objectives of the CFGB continue to align with government priorities?

Conclusion – The objectives of the CFGB align with government priorities.

Supporting Evidence – One of the 2009 long-term priorities of the Government of Canada is Rebuilding the Canadian Forces. In the Canada First Defence Strategy, the government notes that given the demographic challenges that will be facing the Canadian workforce in the coming decades, Defence will continue to strive for excellence by recruiting and retaining quality candidates that reflect the face of Canada.^{Footnote 4} Furthermore, it notes that People are Defence’s most important resource. Both the Department and the Forces rely heavily on the work and expertise of dedicated personnel to ensure the operational effectiveness of the military.^{Footnote 5} A few interviewees (4 of 15), including the CDS , felt the CFGB plays a direct role, and many (10 of 15) felt they play an indirect role in this priority in part by: making the CF a better employer by improving policies, the administration and the grievance process (8 of 15); helping to increase morale (4 of 15); and, increasing retention (4 of 15).

3.1.3 Do the CFGB’s activities align with federal roles and responsibilities?

Conclusion – The federal government mandate clearly includes the creation and maintenance of an armed force. The CFGB supports this and is therefore aligned with federal roles and responsibilities. However, awareness activities and follow-up on recommendations on systemic issues are within the legislative intent only to the extent that they help the CFGB deliver on its mandate of review of military grievances.

Supporting Evidence – Section 14 of the NDA identifies the role of maintaining the armed forces as a federal government role. As defined in section 29 of the NDA, the CFGB’s is to support the CF through its role in the grievance process.

However, the NDA and the QR&O define the role of the CFGB as limited to providing recommendations on grievances. Therefore, the CFGB objectives related to increasing awareness and understanding of the grievance process are within the legislative intent only so long as they support the providing of recommendations. The fact that the CFGB has received requests and invitations to speak at CF gatherings regarding their role, as conveyed by CFGB management representatives, is evidence of this support. However, while the CFGB sees the identification of systemic issues as within their mandate, further follow-up and surveying of actions taken by the CF in response to CFGB recommendations is beyond their purview, except to the extent that it aids them in developing recommendations for future F&Rs.

3.2 Performance (Effectiveness)

The following section seeks to determine the extent to which the CFGB and the Review of Military Grievances Program have achieved their intended results over the past five years.

3.2.1 Does the CDS have an increased awareness and understanding of systemic issues?

Conclusion – The CDS has an increased awareness and understanding of systemic issues arising from grievances referred to the CFGB. Just over 20 recommendations on systemic issues are communicated annually and the CDS agrees that they are useful and about 60% merit further study.

Supporting Evidence – Each year the CFGB communicates approximately 20 recommendations on systemic issues to the CDS . On average, 75% of these are related to interpretation and application of regulations, policies and guidelines. In interviews, the CDS confirmed that the recommendations on systemic issues communicated to him are very useful. While some decisions are pending on 2008 and 2009 recommendations, between 2005 and 2007 the CDS agreed that between 54.5% and 66.7% of recommendations on systemic issues merited further action.

Table 2. CFGB Referred Recommendations on Systemic Issues and CDS Response
Indicators	2005	2006	2007	2008	2009
# of recommendations on systemic issues identified by the CFGB which are communicated to the CDS	25	21	22	22	20
# of times the CDS agreed a systemic issue recommendation merited action	15	14	12	6^{Table note *}	n/a^{Table note **}

Table footnote *

awaiting CDS Decision on 7 cases

Return to table footnote * referrer

Table footnote **

awaiting CDS Decisions on 20 cases

Return to table footnote ** referrer

When asked about barriers to communication of recommendations on systemic issues, no single barrier seemed prominent. Internal CFGB staff noted, as a barrier, their difficulty in tracking the outcome of the CF actions as follow-up to the Board’s recommendations on systemic issues. This impedes the evaluation of the effect their recommendations have on the effectiveness and administration of the CF.

3.2.2 Does the CFGB assist the CDS in rendering decisions on grievances?

Conclusion – The CDS agrees that the CFGB F&Rs assist him in rendering decisions. He fully agreed with them more than half of the time and in the 7% of F&Rs where he disagreed, he nonetheless perceived them as useful. Only 3% of the time, the CDS disagreed based on what he considered to be errors in law.

Supporting Evidence – Since 2005 the CDS fully agreed with 55.6% of the CFGB F&Rs. While he disagreed with on average 7.2% of the F&Rs, only 2.7% of the time was this due to what the CDS considers to be an error in fact or law.

Table 3. CDS Acceptance of F&Rs
% of F&Rs with which the CDS . . .	2005	2006	2007	2008	2009	Avg
Fully Agrees	70.1%	42.6%	59.8%	50.0%	55.1%	55.6%
Partially Agrees	19.4%	10.3%	12.1%	13.8%	6.5%	12.8%
Disagrees	6.9%	7.4%	5.6%	8.5%	8.4%	7.4%
Disagrees due to what he considers to be errors in law or fact	0.7%	5.1%	3.7%	1.5%	n/a	2.7%^{Table note *}
Has not decided (withdrawals, pending, etc.)	3.5%	39.7%	22.4%	27.7%	29.9%	24.2%

Table footnote *

Average for 2005–2008

Return to table footnote * referrer

All decision makers (3 of 3) agree that even when they do not accept the CFGB recommendation, the CFGB work is of value to them. Furthermore, most interviewees (11 of 12) and all decision makers, including the CDS , agree the F&Rs are useful. They noted that the F&Rs present the reasoning behind decisions (4 of 10), and the analysis was comprehensive (2 of 10) and of high quality (2 of 10). Two of the three counsels for grievors noted that the F&Rs gave insights into decisions and were useful in explaining to their clients how the Final Authority will, or may, rule on their cases.

3.2.3 Has the work of the CFGB contributed to an increase in stakeholders’ awareness and understanding?

Conclusion – The extent of increased awareness and understanding due to outreach activities has not been tested, but stakeholders perceive increased awareness. For the 32 cases where there has been an informal resolution, clearly there has been an increased awareness and understanding.

Supporting Evidence – The CFGB is undertaking outreach activities so as to inform stakeholders about their role in the CF grievance process and share information about precedence gained through the review of past cases. It does this through a number of means including publishing cases on their Web site, producing an eBulletin, publishing a newsletter (Perspectives) which details recommendations on systemic issues, adding inserts to the CF-wide newspaper The Maple Leaf, as well as visiting bases.

Over the five year evaluation period the number of case summaries published on the Web site has more than doubled from 89 to 201. Similarly, the number of case summaries published in the e- bulletin has quadrupled over the past four years from 3 to 12. The CFGB has also been able to reach more individuals as the number of eBulletin subscribers has risen from 508 in 2008 to 590 in 2009. Members and staff of the CFGB have done 17 base visits and 16 presentations within Canada on the CFGB’s role in the CF grievance process, including providing examples of past cases. Unfortunately there have been no studies or reviews on the effectiveness or usefulness of these outreach activities.

Nonetheless, many interviewees (10 of 15) perceive an increase in awareness and understanding of the CF grievance process, regulations, policies and guidelines. (Three others ‘did not know’.) One third (5 of 15) specifically mentioned outreach and communication tools as useful in raising awareness. However, some (4 of 15) noted that individuals only seek out information once they have a grievance. Publications were perceived as having more impact than base visits. This may be because the audience of a base visit is defined by who is present, but publications are more widely distributed and are therefore available to any who are interested, regardless of location. A few interviewees and focus group participants were also concerned that outreach activities, especially base visits, may be causing confusion regarding the role of the CFGB and CFGA. For instance, some grievors have sought to specifically have their grievance reviewed and/or adjudicated by the CFGB, ignoring the process which must be followed to determine whether or not a grievance goes to the CFGB and overall authorities. However, it is unclear whether these instances of confusion were generated by base visits, or overall misunderstanding regarding the grievance process.

The CFGB grievor survey results show awareness of the CFGB has increased from 57% to 84% between 2006 and 2009. The most commonly cited mechanisms which alerted grievors to the existence of the CFGB were word-of-mouth (colleagues and COs), individual research, briefings, Web site, professional development, or CF or CFGB publications.

With regards to increasing understanding, many interviewees (11 of 17) agreed that the CFGB’s work has lead to improved interpretation of CF policies, regulations, and guidance. (Two others did not know or did not provide a clear response.) However, for the interviewees who act as IAs, some disagreed (3 of 7). Therefore those who would be most responsible for interpreting these policies are not seeing the benefit of F&Rs. Of those who did agree, they noted that the CFGB raises concerns about systemic issues (4 of 17). The influence of recommendations on systemic issues is confirmed by that fact that of the 25 systemic recommendations from 2009, 20 dealt with interpretation and application of regulations, etc. The CDS agreed with 70% of these. Several interviewees (3) provided examples of where the CFGB had improved interpretation and/or application of CF regulations, policies or guidelines.

When the CFGB produces its F&Rs a copy is sent both to the CDS and the grievor. Every year about six grievors withdraw their grievance at this point because the F&R delineates an interpretation which provides the grievor with a clear explanation and closure. In these instances, the grievor clearly has an increased awareness and understanding of the CF policies, regulations and guidelines. A further twelve or so cases are resolved through informal resolution, whereby the grievor and the CF work towards an understanding after the issuance of the F&R and prior to the CDS decision.

Table 4. Informal Resolutions & Withdrawals
# of Grievances . . .	2005	2006	2007	2008	2009	Total
Withdrawn due to F&Rs which delineate interpretation (withdrawn after F&R, but before CDS decision)	5	7	8	7	5	32
Resolved through informal resolution (completed after issuance of the F&R but prior to CDS decision)	13	12	6	21	8	60

3.2.4 Have regulations, policies, guidelines and the grievance process been improved as a result of the CFGB’s work?

Conclusion – There is no clear evidence that the CFGB’s work has led to improvements to regulations, policies, guidelines and the grievance process. Although, perceptions are that it has.

Supporting Evidence – Every year the CF conducts a “Your Say” survey among members. The results do not suggest that there has been any improvement in policies, given that the proportion of those who believe that policies are equally fair to everyone has not changed over the evaluation period. In fact more respondents disagree that policies are fair than agree (38.2% versus 36.8%).

Just over half of interviewees (10 of 19) agreed that the work of the CFGB had led to improvements (5 disagreed, 3 were not sure, and 1 did not provide a clear response). However, half of those who agreed (5 of 10) noted it is difficult to attribute a policy change to any single factor or questioned whether the change would have occurred regardless of the CFGB’s work. A few (3 of 19) noted that it was not easy to change policies as they often affect the whole of the public service and are within the purview of the Treasury Board.

3.2.5 What impact has the work of the CFGB had on confidence levels in the grievance process?

Conclusion – There is no clear evidence that confidence levels in the grievance process are increasing, nor is it clear whether the CFGB is having an impact on this confidence. Nonetheless, grievors seem satisfied with the fairness, transparency and equity of the process, although less satisfied with timeliness.

Supporting Evidence – The “Your Say” survey results suggest that CF members experienced a reduction in confidence levels in the grievance process, especially between fall 2006 and fall 2007. When asked whether CF members who submit a grievance are likely to obtain justice, on average 39.0% agreed, 37.5% were neutral and 23.5% disagreed prior to fall 2006. However, in fall 2007 those in agreement dropped to 30.4% and those who were neutral rose to 45.0%. These reduced levels remain constant for the remainder of the evaluation period. This drop in agreement and rise in neutral response between fall 2006 and 2007 is consistent across all questions pertaining to justice and fairness, and raises the question of whether something adverse occurred in that period.

Figure 2. “Your Say”: Grievors are likely to obtain justice

When asked whether “methods for resolving complaints in the CF are fair and unbiased” overall 36.8% agreed and 27.7% disagreed. As with the previous question, there was a notable 7.9% drop in agreement between fall 2006 and fall 2007, but since then agreement has begun to return to previous levels.

Complaint resolution is fair and unbiased — Figure 3. “Your Say”: Complaint resolution is fair & unbiased

Stakeholders Interviews

Half of the interviewees (7 of 14) indicated an increase in confidence and satisfaction in the CF grievance process. A few interviewees cited improved timeliness (3), removal of adjudication from the chain of command (2) and the clarified, better understood grievance process (2) as reasons for increased confidence. Interviewees were split as to whether the CFGB had contributed to the change (5 said ‘yes’, 4 said ‘no’, 1 said ‘don’t know’).

Interviewees were also split as to whether there had been an improvement in their satisfaction with the fairness, equity, transparency, and timeliness of the F&Rs. Those who noted improvements (5 of 9) cited improved procedures (3), and timeliness (3). Those who were less impressed (4 of 9) noted that all the current Board Members have previous military experience and were of the view that this lessened the Board’s independence (3).

CFGB’s Grievors Survey

The Grievors Survey found that grievors were generally satisfied with fairness, as 79.6% agreed or strongly agreed that they were provided with appropriate disclosure of information. Similarly they are satisfied with equity and transparency, as 68.4% agreed or strongly agreed that their grievance had been reviewed in a fair and unbiased manner. However, less than half (49.5%) agreed that their grievance was handled in a timely manner. Note that the majority of respondents referred to the timeliness of the whole process meaning from the time the grievance was filed within the CF to the time of the CDS Decision on their case. No trends were apparent on any of these measures.

3.2.6 Have the F&Rs of the CFGB been implemented and led to improvements in CF conditions of service?

Conclusion – It is not possible to know the extent to which CFGB F&Rs have been implemented, as the CFGB is not made aware of actions in response to F&Rs.

While the CF "Your Say" Survey indicates a certain level of dissatisfaction amongst CF members relating to policies, interviewees believe that F&Rs of the CFGB have lead to improvements in the conditions of service. It is difficult, however, to attribute policy change to any one factor.

Supporting Evidence – While the CDS must provide feedback to the CFGB regarding his decisions relating to F&Rs, the CFGB is not systematically made aware of actions taken by the CF Chain of Command in response to these F&Rs. Furthermore, it is not within the CFGB’s mandate to follow-up on its recommendations on systemic issues, although they do make an effort to do so, considering it a best practice.

Stakeholders perceive that the CFGB plays a role in rebuilding the CF, and thus the conditions of service, by making the CF a better employer, increasing morale and retention. As mentioned in section 3.1.2, overall, two thirds of interviewees (10 of 15) felt that the CFGB plays an indirect role in the rebuilding of the CF, while some others (4 of 15, including the CDS) felt the CFGB plays a direct role. The most commonly cited reason was that the CFGB makes the CF a better employer (8) through improving policies, the administration and the grievance process. Some others (4) noted that the CFGB helps to increase morale, while others (4) thought the CFGB helped to increase retention.

During the course of the evaluation, GCS examined the extent to which the CFGB has: improved regulations, policies, guidelines and the grievance process; helped in the administration of the affairs of the CF; and impacted on the morale of CF members. As mentioned in section 3.2.4 above, there is no clear evidence that the CFGB’s work has lead to improvements to regulations, policies, guidelines and the grievance process. Although, perceptions are that it has.

With regards to whether or not the CFGB has helped in the administration of affairs of the CF, most interviewees (12 of 15) agreed that the CFGB has helped in the administration of the affairs of the CF. They noted that the CFGB F&Rs raise awareness of areas where policy changes may be needed (5) and help confirm interpretation of policies (2), allowing IAs to be more confident in their decisions. The Web site and publications also help IAs and potential grievors in deciding on how to proceed when issues arise.

All interviewees were satisfied with the equity, transparency and fairness of the CFGB. Furthermore, many interviewees (13 of 18) agreed that the CFGB contributes to fairness, equity and transparency of the grievance process overall. Many (9 of 18) cited their objectivity and impartiality since they are outside of the chain of command.

Finally, with respect to the impact on the morale of CF members, background documents, quoting the then CDS and MND, suggest that the outputs of the CFGB have benefitted CF members’ confidence and morale. CDS , General Rick Hillier noted that the CFGB was able to elevate the confidence of military members in regard to the grievance process and to target the areas where human resource policy should be reinforced, thereby by contributing to the improved morale and efficiency of the CF.^{Footnote 6} In his report, Chief Justice Lamer notes that effectively responding to grievances is critical to maintaining a high morale among CF members.^{Footnote 7} Most interviewees agreed (14 of 18), including all of the counsel. They noted that the CFGB offers an opportunity for redress (6), provides closure and removes uncertainty (4).

3.2.7 What factors contributed to, or detracted from, the CFGB completing its work and reaching its objectives?

Conclusion – Barriers to the achievement of objectives are: resistance on the part of the chain of command to input from an outside organization; and the rotational nature of the CF and the subsequent need to rebuild relationships. With regards to achieving fairness, some suggest that the CFGB should have input on all grievances.

Supporting Evidence – Some interviewees (7 of 18) and the focus group participants cited resistance of the CF chain of command to an outside organization passing judgment on them. While some (2 of 7) felt this was more common at the outset of the CFGB, the majority (5 of 7) considered this still to be an issue today.

Some internal interviewees (2 of 5) and the focus group participants noted the rotational nature of the CF as a barrier as it requires the CFGB to rebuild relationships and awareness every two years.

To achieve fairness for all grievors, a few (2 of 18) suggested that the CFGB should have input on all grievances. While others were aware of this perceived barrier, but did not view it as a barrier (2 of 18).

3.3 Performance (Efficiency & Economy)

Under the 2009 Evaluation Policy, efficiency is defined to be maximizing the outputs produced with a fixed level of inputs or minimizing the inputs used to produce a fixed level of outputs (paraphrase); and economy is defined to be “minimizing the use of resources [...] to achieve expected outcomes”.^{Footnote 8} Therefore, for the purposes of the Evaluation Policy, these elements of performance are demonstrated when:

outputs are produced at minimum cost (efficiency); and,
outcomes are produced at minimum cost (economy).

It has been determined that outputs related to F&Rs and outreach activities can be quantified, but the extent of the outcome achievement has not been captured. Following is a discussion of the cost of producing the outputs and, to the extent possible, outcomes.

3.3.1 Are there more effective and efficient means of achieving the CFGB’s objectives?

Conclusion – The average cost per F&R, when considering the CFGB budget overall, is $50,141, which seems significant. However, this may be the cost required to maintain an independent organization. Most stakeholders suggest there are better ways to achieve the outcomes, including removing some of the duplication of effort between the CFGB and the CFGA.

Supporting Evidence – The average cost per F&R, when considering the CFGB budget overall is $50,141. When including only costs associated with the Review of Military Grievances Program, as per the PAA, the cost is $25,807. The actual spending both overall and on Review of Military Grievances remained relatively constant (+/- $200K), over the five year period of the evaluation. Many stakeholders (10 of 14) perceive the program elements to be adequate. Some interviewees (4 of 14) question whether the program cost is too high given the outputs. It was noted that $6 million is a high price to pay for 150 grievances.

The average time to process a grievance at the Board dropped each year, from 417 days in 2005 to 199 days in 2009, in spite of an increase in overall complexity of the cases. The proportion of single issue, simple cases has dropped from 42.5% in 2006 to 23.6% in 2009, while the proportion of moderate cases has increased from 29.1% to 50.0%. Proportionally, complex cases have remained fairly stable.

Virtually all stakeholders (18 of 19) suggested that there are better ways to achieve CFGB outcomes. Many (13 of 19) noted the duplication of effort between the CFGB and the CFGA. Most of these suggested a restructured grievance process removing the duplication, such as the Principled Approach^{Footnote 9} which is currently being discussed within the CF. Focus group participants agreed.

A few interviewees (3 of 19) hoped for greater communication of F&Rs so as to provide lessons learned and best practices to possible grievors or IAs and prevent grievances. Others (3 of 19) felt the grievance process as a whole could be improved if alternative dispute resolution and mediation were encouraged. Finally, a couple of others (2 of 19) felt the CDS should be given the authority to approve financial rewards. These last two measures are beyond the scope of the evaluation and the CFGB’s control, but are nonetheless provided here for information.

3.3.2 Is the CFGB delivering this program efficiently? How could this be improved?

Conclusion – There are some areas where the CFGB seems to be working efficiently and has made improvements, such as through its review of its internal processes and consistent reduction in time to process a grievance. However, there are other indications that further efficiencies may be possible, such as the proportion of funds which are spent on internal services and the average utilization rate of grievance officers.

Supporting Evidence – The CFGB has been regularly examining its processes in an effort to make them as efficient as possible. Over the past ten years it has had 22 versions of its grievance process and it seems the changes have improved efficiency. Efficiency has been helped by the implementation of a very capable knowledge management system, WebCIMS, which helps track grievances and past precedents. Improvements to the process have meant that over the past five years the time to process a grievance has consistently dropped, going from 417 days in 2005 down to 199 days in 2009. The CFGB has also been progressively getting better at meeting its six month turn-around goal. In 2005 only 4.2% of F&Rs met the goal, but by 2009 41.1% of grievances were resolved within six months.

Despite the fact that the time to process a grievance within the CFGB has been going down, stakeholders still see room for improvement. Many interviewees (13 of 19) and most CF stakeholders (10 of 11) had suggestions as to how the CFGB process could be further improved. However, many of these reflected inefficiencies in the larger grievance process, as defined in Figure 1, such as the duplication of review structures and analysis or the need for alternative dispute resolution mechanisms. These echoed the comments noted in 3.3.1 regarding improved mechanisms for achieving outcomes, particularly the need to reduce duplication. However, there were specific areas for increased efficiency internal to CFGB which were mentioned such as the need to better define the roles of the Grievance Officers versus the CFGB lawyers.

The actual spending of the CFGB is in line with planned spending, as actual spending has remained within 11.3% of planned spending for both overall spending and for the Review of Military Grievances Program during the five year evaluation period. As noted in 3.3.1, spending has been relatively constant.

Over the five year period of the evaluation, the average cost per F&R has gone up to a high in 2007 of $31,075, before falling again to $22,056 in 2009 (see Figure 4). The number of outreach tools has been increasing, but these are relatively new so no trends can be reported. Other outcome information, such as improvements to the administration of the CF or to policies, is not available.

Cost per Findings and Recommendations — Figure 4. Cost per F&R

CFGB spending and FTEs, as reported in the DPRs, seems to be overly weighted towards internal services. Over the five year evaluation period, the Board spent on average of 48.8% of its budget on internal services. This is much higher than the 33% which is typically allocated in Treasury Board submissions for new programs. Part of this is attributed to the fact that the CFGB does not use Activity Based Costing, allocating proportions of rent or IT services to the Review of Military Grievances Program as appropriate. Instead, all costs beyond the salaries of those working on the grievance program are attributed to internal services. Internal services FTEs represent 36.7% of CFGB staff. As a small agency it is not surprising that internal services represent a larger proportion of staff than the 20% benchmark. However, this is an area where further efficiencies might be had.

In order to track cost per grievance, grievance officers and many of the other staff of the CFGB track their time against grievances. This enables management to assess staff utilization against target utilization rates. The average utilization of grievance officers has dropped off over the evaluation period by 9.8% (from 46.0% in 2005 to 36.2% in 2009) and is well below the target utilization rate of 60.0%. Anecdotally, it has been reported that some grievance officers were not reporting their time. However, even when delinquencies in reporting are accounted for, utilizations still remain below target for 2008 and 2009. It has been suggested that this is due to a drop in grievances. Data regarding the utilization rates of other staff was either not available or no benchmarks were provided.

4. Summary and Recommendations

4.1 Relevance

There is a continued need for the Review of Military Grievances Program at the CFGB given that external review of military grievances is legislated. In addition, the mandate of the CFGB is to provide recommendations to the CDS , and the CDS perceives a continued need. This is further supported by the fact that the CFGB has discretionary files referred to them, suggesting that others find the CFGB valuable as well. However, as noted in the Frecker report, there is considerable duplication in the work of the Canadian Forces Grievance Authority (CFGA) and the CFGB, in relation to analysis of the grievances.

The objectives of the CFGB align with government priorities. One of the 2009 long-term priorities of the Government of Canada is Rebuilding the Canadian Forces. The Canada First Defence Strategy also emphasizes recruiting and retaining quality CF members. Interviewees noted that the CFGB plays an indirect role in this priority by: making the CF a better employer improving policies, the administration and the grievance process (8 of 15); helping to increase morale; and, increasing retention. However, awareness activities currently being conducted by the CFGB and follow-up on recommendations on systemic issues are within the legislative intent only to the extent that they support the provision of recommendations.

Recommendation # 1: The CFGB should examine its activities to ensure they align with its mandate.

The CFGB activities align with federal roles and responsibilities, since the federal government mandate clearly includes the creation and maintenance of an armed force. The CFGB supports this and is therefore aligned with federal roles and responsibilities.

4.2 Performance (Effectiveness)

The CDS has an increased awareness and understanding of systemic issues arising from grievances referred to the CFGB. Just over 20 recommendations on systemic issues are communicated annually and the CDS agrees that they are useful and about 60% merit further study. When asked about barriers to communication of recommendations on systemic issues, no single barrier seemed prominent to interviewees.

The CDS agrees that the CFGB F&Rs assist him in rendering decisions. He fully agreed with them more than half of the time and in the 7% of F&Rs where he disagreed, he nonetheless perceived them as useful. Only 3% of the time did he disagree based upon what he considers to be errors in law. Furthermore, most interviewees and all decision makers agree the F&Rs are useful. They noted that the F&Rs present the reasoning behind decisions, and the analysis was comprehensive and of high quality.

Stakeholders perceive an increased awareness on the part of CF members of the CF grievance process and related regulations, policies and guidelines; however, the extent to which CFGB outreach activities have increased awareness and understanding has not been tested. For the 32 cases where there has been an informal resolution, clearly there has been an increased awareness and understanding of the CF policies, regulations and guidelines.

Recommendation #2: The CFGB should evaluate the effectiveness of the various outreach activities

There is no clear evidence that the CFGB’s work has lead to improvements to regulations, policies, guidelines and the grievance process, although perceptions are that it has. Interviewees admitted, however, that it is difficult to attribute policy change to any single factor and some questioned whether the change would have occurred regardless of the CFGB’s work. Similarly, there is no clear evidence that confidence levels in the grievance process are increasing, nor is it clear whether the CFGB is having an impact on this confidence. Nonetheless, grievors seem satisfied with the fairness, transparency and equity of the process, although less satisfied with timeliness.

It is not possible to know the extent to which CFGB F&Rs have been implemented, as the CFGB is not made aware of actions in response to F&Rs.

Barriers to the achievement of CFGB objectives are: resistance on the part of the chain of command to input from an outside organization; and the rotational nature of the CF and the need to rebuild relationships. With regards to achieving fairness, some suggest that the CFGB should have input on all grievances.

4.3 Performance (Efficiency & Economy)

There are some areas where the CFGB seems to be working efficiently and has made improvements. The CFGB has been regularly examining its processes in an effort to make them as efficient as possible. Improvements made to the process have meant that over the past five years the time to process a grievance has consistently dropped, going from 417 days in 2005 down to 199 days in 2009. The CFGB has also been progressively getting better at meeting its six-month turn-around goal. In 2005 only 4.2% of F&Rs met the goal, but by 2009 41.1% of grievances were resolved within six months.

However, in spite of the improvements already made to the process, stakeholders still suggest there are better ways to achieve the outcomes, particularly with regard to removing some of the duplication of effort between the CFGB and the CFGA.

Recommendation # 3: The CFGB should continue to work with the CFGA to streamline the process and remove duplication.

Recommendation # 4: The CFGB should examine the allocation of spending and FTEs on internal services versus the review of grievances. This includes ensuring that the CFGB has the best possible information available regarding the cost of operations.

Appendix A: Logic Model

Logic Model for the Canadian Forces Grievance Board
(Revised May 18, 2010)

Appendix B: Evaluation Matrix

Evaluation questions	Ind. #	Indicators	Data source	Methodology
Relevance
1. Is there a continued need for an external review of military grievances? (TBS core issue 1)	1.1	Extent to which stakeholders perceive gaps would exist within the CF grievance process without the CFGB	Stakeholders	Interviews
	1.2	Extent to which stakeholders perceive there are other models which fulfill the role of the CFGB.	Stakeholders	Interviews
	1.3	Have legislation reviews validated the continuing need for an independent organisation, external to the CF and DND, to deal with grievances of CF members'	Director of Legal Service; Foundational documents	Interview Document review
	1.4	Trend in # of individual cases where the CFGB agreed with the grievor (difference of opinion from the chain of command)	Program data	Data analysis
	1.5	Trend in # of systemic issues raised by CFGB	Program data	Data analysis
	1.6	Trend in proportion of referred files which are discretionary	Program data	Data analysis
2. Do objectives of the CFGB continue to align with government priorities? (TBS core issue 2)	2.1	Concurrence over time of CFGB objectives and government priorities.	Foundational documents	Document review
	2.2	Alignment of CFGB objectives with the legislative intent	Foundational documents	Document review
	2.3	Extent to which stakeholders perceive the CFGB contributes to the "Rebuilding the CF" priority	Stakeholders	Interviews Document review
	2.4	Extent to which activities of CFGB align with government values of equity, transparency and fairness	Stakeholders Foundation documents Survey results	Interviews Document review Data analysis
3. Do the CFGB's activities, outputs and outcomes align with federal roles and responsibilities? (TBS core issue 3)	3.1	Degree to which there is alignment between the mandates of the CFGB and the federal government	Foundational documents	Document review
Performance (effectiveness, efficiency and economy)
Effectiveness - Assessment of progress towards expected outcomes
Immediate
4. To what extent does the CDS have an increased awareness and understanding of systemic issues arising from grievances referred to the CFGB?	4.1	Trend in number of systemic issues identified by the CFGB which are communicated to the CDS	Program data; Reports to CDS	Data analysis Document review
	4.2	Extent to which CDS perceives the identified issues as being useful/valid	CDS	Interviews
	4.3	Presence of obstacles in communication between CFGB and CDS regarding systemic issues	CFGB & CDS	Interviews
	4.4	Number of times the CDS agreed a systemic issue recommendation merited action	Program data	Data analysis
5. To what extent does the CFGB assist the CDS in rendering decisions on grievances?	5.1	Proportion of F&Rs with which the CDS disagrees on the basis of errors in law or fact	CDS Decisions	Data analysis
	5.2	Proportion of F&Rs with which the CDS fully agrees	CDS Decisions	Data analysis
	5.3	Extent to which stakeholders perceive F&Rs to be useful	Stakeholders	Interviews
	5.4	Trends in time required to process a grievance at the Board	CFGB case management and tracking system	Data analysis
6. To what extent has the work of the CFGB contributed to an increase in stakeholders' awareness and understanding of the CF grievance process and related regulations, policies and guidelines affecting CF members?	6.1	Level of accessibility of case summaries for the CF and its members	# of reports & publications Analysis of "types" of participants	Data analysis
	6.2	Grievors' awareness of the existence of an external review of military grievances prior to their grievance	Grievor survey	Data analysis
	6.3	Mechanism which alerted the grievor to the existence of the CFGB and the grievance process	Grievor survey	Data analysis
	6.4	Trend in reach of communications tools - interest in case summaries and tools on Web site - perception of usefulness of publications	Web site hits eBulletin subscribers Perspectives subscribers	Data analysis Option to add survey question in eBulletin
	6.5	Extent to which stakeholders perceive there is an increased awareness and understanding of the CF grievance process, regulations, policies and guidelines	Stakeholders	Interviews
	6.6	Extent to which stakeholders perceive the communication tools as being useful/valid	Stakeholders	Interviews
	6.7	Extent to which stakeholders perceive an improvement in the interpretation/ understanding and application of regulations, policies & guidelines	Stakeholders	Interviews
	6.8	# of times a grievance is withdrawn due to F&Rs which delineate interpretation (informal resolutions)	Program data	Data analysis
Intermediate
7. As a result of the CFGB's work, to what extent have regulations, policies, guidelines and the grievance process been improved?	7.1	Extent to which stakeholder perceive the changes to regulations, policies, guidelines and the grievance process as an improvement.	Stakeholders	Interviews
	7.2	Where the CDS agreed a systemic recommendation merited further action, perceptions that work led to improvements	Stakeholders	Interviews
8. What impact has the work of the CFGB had on confidence levels in the grievance process?	8.1	Trend in % of stakeholders satisfied with the fairness, timeliness, equity and transparency of the Board's F&Rs	Grievor survey Stakeholders	Data analysis Interviews
	8.2	Extent to which stakeholders perceive changes in confidence and satisfaction in the grievance process - perceived correlation between variance and work of the CFGB	Stakeholders	Interviews
Strategic outcome
9. To what extent have the effectiveness of the CF & morale of its members been enhanced by the CFGB external review of military grievances in accordance with the government values of equity, transparency & fairness? (PAA-PMF SO)	9.1	Extent to which stakeholders perceive the CFGB outputs (e.g. F&Rs, Board publications) as helpful in the administration of the affairs of the CF.	Stakeholders	Interviews
	9.2	Extent to which stakeholders perceive the CFGB outputs (e.g. F&Rs, Board publications) benefit CF members’ confidence & morale.	Stakeholders	Interviews
	9.3	Extent to which stakeholders are satisfied with the independence, fairness, equity & transparency of the CFGB.	Stakeholders	Interviews
	9.4	Extent to which stakeholders perceive the CFGB contributes to fairness, equity and transparency within the CF.	Stakeholders	Interviews
10. What factors contributed or detracted the CFGB from completing its work and reaching its objectives?	10.1	Identification of factors that contribute to/detract the CFGB from reaching/accomplishing the following: - outputs - immediate outcomes - intermediate outcomes - strategic outcomes	Program managers Stakeholders	Interviews Focus Group
Efficiency and Economy - assessment of resource utilization in relation to the production of outputs and progress towards expected outcomes
11. Are there more effective and efficient means of achieving the CFGB's objectives?	11.1	Extent to which stakeholders perceive the program elements as adequate and appropriate to achievement of intended results, compared to alternative design/delivery approaches	Stakeholders	Interviews Focus Group
	11.2	Comparison of trends over the years in time required, average cost per grievance	Program data	Data analysis
	11.3	Extent to which stakeholders perceive the CFGB could use alternative methods/ processes	Stakeholders	Interviews Focus Group
12. Is the CFGB delivering this program efficiently? How could this be improved?	12.1	Extent and nature of stakeholder perceptions of how the program efficiency could be improved	Stakeholders	Interviews Focus Group
	12.2	Evidence of areas for improved efficiency	Program data/cost per grievance	Data analysis
	12.3	Assessment of level of outputs given available inputs ($, FTEs)	Program data	Data analysis
	12.4	Assessment of distribution of resources relative to strategic priorities and operational needs	Program data	Data analysis
	12.5	Trend in cost to deliver	Program data	Data analysis
	12.6	Trend in timelines of recommendations	Program data	Data analysis

Appendix C: List of Documents Reviewed

Title	Author	Year
National Defence Act s.29	GoC	1998
Queen's Regulations & Orders for the CF	GoC	1998
Lamer Report	Lamer, Antonio	2003
RMAF	CFGB	2004
CFGB Mission, Vision & Values	CFGB	2009
RPP	CFGB	2009
DPR 2008-09	CFGB	2009
DPR 2007-08	CFGB	2008
DPR 2006-07	CFGB	2007
DPR 2005-06	CFGB	2006
DPR 2004-05	CFGB	2005
2009 Government Priorities	GoC	2009
Canada First Defence Strategy	GoC	2009
Frequently Asked Questions	CFGB	2009
Annual Report	CFGB	2008
Annual Report	CFGB	2007
Annual Report	CFGB	2006
Annual Report	CFGB	2005
eBulletin vol.1	CFGB	2009
eBulletin vol.2	CFGB	2009
Perspectives vol.1, no.1	CFGB	2008
Perspectives vol.1, no.2	CFGB	2009
It's the militarization of civilian oversight organizations	Drapeau, Michel	2009
Review of Canadian Forces Grievance Board Internal Case Management Processes	Frecker, John	2008

Appendix D: Interview Guides

Interview Guide – CDS & DGCFGA

Government Consulting Services (GCS) has been asked to conduct the evaluation of the Canadian Forces Grievance Board (CFGB) in response to the Treasury Board Secretariat requirement that a comprehensive review be conducted every five years. Thus an evaluation is being conducted that will focus on assessing the continued relevance and performance of the CFGB as required by the new federal government Policy on Evaluation. As part of the evaluation, GCS is conducting interviews with key stakeholders involved in the CFGB's work. The following questions will serve as a guide for the interview. The goal of the interviews is to gather information about the CFGB's continued relevance, the impacts of its work (performance), as well as possible improvements.

If you have any questions please do not hesitate to contact Kathy Gerber, GCS Project Manager at 613-995-1142.

Background information on the CFGB

The Canadian Forces Grievance Board was created in accordance with amendments made to the National Defence Act in 1998 as a result of studies and inquiries into solutions for perceived inequities with the military's complaint resolution methods. The amendments focused on simplifying and improving the timeliness of the grievance process and the Board received its regulatory authority and officially began operations on June 15, 2000.

The objective of the CFGB is to enhance the effectiveness of the Canadian Forces (CF) and the morale of its members through its external reviews of military grievances in accordance with the values of equity, transparency and fairness. It hopes to achieve this by enhancing confidence in the grievance process and promoting improvements to the regulations, policies, guidelines and overall grievance process.

Introduction

1. Please briefly describe how the CFGB's work impacts your organization and your own work.

2. How long have you been affected / influenced by its work?

Performance - Effectiveness

3. How would you describe the CFGB findings and recommendations (F&Rs) and their impacts on your organisation?

a. Are they useful? Valid? Please explain why or why not.

b. In instances where you have not accepted the recommendation of the CFGB, was their work of value to you?

4. In your opinion, has the CFGB's work led to improved interpretation and application of the Canadian Forces' (CF) regulations, policies and guidelines? Please explain why or why not.

5. In your opinion, has the work of the CFGB led to improvements to:

a. The regulations, policies, guidelines and grievance process?

b. Other elements impacting the CF and its members?

6. How are recommendations and follow-up actions on systemic issues communicated? Are there barriers that have impeded communication?

7. Has CF members' understanding of the CF grievance process, regulations, policies and guidelines improved in the last five years?

a. Why or why not?

b. If so, do you believe the CFGB played a role in this?

8. Has CF members' confidence in the grievance process improved in the last five years? Why or why not?

a. If so, did the CFGB play a role in this?

9. Overall, have the outputs of the CFGB including F&Rs, publications and outreach activities:

a. Helped in the administration of the affairs of the CF? Why or why not?

b. Benefitted CF members' morale? Why or why not?

c. Contributed to the fairness, equity and transparency of the CF? Why or why not?

Relevance

10. If the CFGB did not exist, would there be gaps in, or other impacts on, the CF grievance process? Why or why not?

11. One of the Government of Canada's priorities is Rebuilding the Canadian Forces. Does the CFGB contribute to this? Why or why not?

12. Do the activities of the CFGB align with the government values of equity, transparency and fairness? Why or why not?

a. Are you satisfied with the independence, equity, transparency, fairness and timeliness of the Boards findings and recommendations?Why or why not?

b. Have you seen a change in any of these aspects over the past five years?

Performance – Efficiency & Economy

13. From your perspective, has the CFGB experienced any barriers or challenges to the achievement of its objectives?

a. Are the program elements (legal, operations, communications, etc.) adequate and appropriate to achieve these?

14. Are there better ways of achieving CFGB outcomes? If so, what alternatives would you suggest?

15. Do you have suggestions as to how the efficiency of CFGB could be improved?

16. Is there something else you would like to add?

Interview Guide – VCDS, CMP & IAs

If you have any questions please do not hesitate to contact Kathy Gerber, GCS Project Manager at 613-995-1142.

Background information on the CFGB

Introduction

1. Please briefly describe how the CFGB's work impacts your organization and your own work.

2. How long have you been affected / influenced by its work?

Performance - Effectiveness

3. How would you describe the CFGB findings and recommendations (F&Rs) and their impacts on your organisation?

a. Are they useful?Valid? Please explain why or why not.

4. In your opinion, has the CFGB's work led to improved interpretation and application of the Canadian Forces' (CF) regulations, policies and guidelines? Please explain why or why not.

5. In your opinion, has the work of the CFGB led to improvements to:

a. The regulations, policies, guidelines and grievance process?

b. Other elements impacting the CF and its members?

6. Has CF members' understanding of the CF grievance process, regulations, policies and guidelines improved in the last five years?

a. Why or why not?

b. If so, do you believe the CFGB played a role in this?

7. Has CF members' confidence in the grievance process improved in the last five years? Why or why not?

a. If so, did the CFGB play a role in this?

8. Overall, have the outputs of the CFGB including F&Rs, publications and outreach activities:

a. Helped in the administration of the affairs of the CF? Why or why not?

b. Benefitted CF members' morale? Why or why not?

c. Contributed to the fairness, equity and transparency of the CF? Why or why not?

Relevance

9. If the CFGB did not exist, would there be gaps in, or other impacts on, the CF grievance process?Why or why not?

10. One of the Government of Canada's priorities is Rebuilding the Canadian Forces. Does the CFGB contribute to this? Why or why not?

11. Do the activities of the CFGB align with the government values of equity, transparency and fairness? Why or why not?

a. Are you satisfied with the independence, equity, transparency, fairness and timeliness of the Boards findings and recommendations?Why or why not?

b. Have you seen a change in any of these aspects over the past five years?

Performance – Efficiency & Economy

12. From your perspective, has the CFGB experienced any barriers or challenges to the achievement of its objectives?

a. Are the program elements (legal, operations, communications, etc.) adequate and appropriate to achieve these?

13. Are there better ways of achieving CFGB outcomes? If so, what alternatives would you suggest?

14. Do you have suggestions as to how the efficiency of CFGB could be improved?

15. Is there something else you would like to add?

Interview Guide – Counsel of Grievors

If you have any questions please do not hesitate to contact Kathy Gerber, GCS Project Manager at 613-995-1142.

Background information on the CFGB

Introduction

1. Please briefly describe how the CFGB's work impacts your organization and your own work.

2. How long have you been affected / influenced by its work?

Performance - Effectiveness

3. How would you describe the CFGB findings and recommendations (F&Rs) and their impacts on your organisation?

a. Are they useful?Valid? Please explain why or why not.

4. In your opinion, has the CFGB's work led to improved interpretation and application of the Canadian Forces' (CF) regulations, policies and guidelines? Please explain why or why not.

5. In your opinion, has the work of the CFGB led to improvements to:

a. The regulations, policies, guidelines and grievance process?

b. Other elements impacting the CF and its members?

6. Has CF members' confidence in the grievance process improved in the last five years? Why or why not?

a. If so, did the CFGB play a role in this?

7. Overall, have the outputs of the CFGB including F&Rs, publications and outreach activities:

a. Benefitted CF members' morale? Why or why not?

b. Contributed to the fairness, equity and transparency of the CF? Why or why not?

Relevance

8. If the CFGB did not exist, would there be gaps in, or other impacts on, the CF grievance process?Why or why not?

9. Do the activities of the CFGB align with the government values of equity, transparency and fairness? Why or why not?

a. Are you satisfied with the independence, equity, transparency, fairness and timeliness of the Boards findings and recommendations? Why or why not?

b. Have you seen a change in any of these aspects over the past five years?

Performance – Efficiency & Economy

10. From your perspective, has the CFGB experienced any barriers or challenges to the achievement of its objectives?

a. Are the program elements (legal, operations, communications, etc.) adequate and appropriate to achieve these?

11. Are there better ways of achieving CFGB outcomes? If so, what alternatives would you suggest?

12. Do you have suggestions as to how the efficiency of CFGB could be improved?

13. Is there something else you would like to add?

Interview Guide – Internal Stakeholders

If you have any questions please do not hesitate to contact Kathy Gerber, GCS Project Manager at 613-995-1142.

Background information on the CFGB

Introduction

1. Please briefly describe your role or involvement with the CFGB.

2. How long have you been working at/with the CFGB?

Performance - Effectiveness

3. In your opinion, has the CFGB's work led to improved interpretation and application of the Canadian Forces' (CF) regulations, policies and guidelines? Please explain why or why not.

4. In your opinion, has the work of the CFGB led to improvements to:

a. The regulations, policies, guidelines and grievance process?

b. Other elements impacting the CF and its members?

5. How are recommendations and follow-up actions on systemic issues communicated? Are there barriers that have impeded communication?

6. Has CF members' understanding of the CF grievance process, regulations, policies and guidelines improved in the last five years?

a. Why or why not?

b. If so, do you believe the CFGB played a role in this?

7. Overall, have the outputs of the CFGB including F&Rs, publications and outreach activities:

a. Helped in the administration of the affairs of the CF? Why or why not?

b. Benefitted CF members' morale? Why or why not?

c. Contributed to the fairness, equity and transparency of the CF? Why or why not?

Relevance

8. If the CFGB did not exist, would there be gaps in, or other impacts on, the CF grievance process?Why or why not?

9. One of the Government of Canada's priorities is "Rebuilding the Canadian Forces." Does the CFGB contribute to this? Why or why not?

10. Do the activities of the CFGB align with the government values of equity, transparency and fairness? Why or why not?

a. Are you satisfied with the independence, equity, transparency, fairness and timeliness of the Boards findings and recommendations? Why or why not?

Performance – Efficiency & Economy

11. From your perspective, has the CFGB experienced any barriers or challenges to the achievement of its objectives?

a. Are the program elements (legal, operations, communications, etc.) adequate and appropriate to achieve these?

12. Are there better ways of achieving CFGB outcomes? If so, what alternatives would you suggest?

13. Do you have suggestions as to how the efficiency of CFGB could be improved?

14. Is there something else you would like to add?

Appendix E: Focus Group Guide

Government Consulting Services (GCS) has been asked to conduct the evaluation of the Canadian Forces Grievance Board (CFGB) in response to the Treasury Board Secretariat requirement that a comprehensive review be conducted every five years. Thus an evaluation is being conducted that will focus on assessing the continued relevance and performance of the CFGB as required by the new federal government Policy on Evaluation. As part of the evaluation, GCS is conducting a focus group to gather the perceptions of program staff as to the effectiveness, efficiency and economy of the CFGB processes.

If you have any questions please do not hesitate to contact Kathy Gerber, GCS Project Manager at 613-995-1142.

Questions

1. What factors contributed to, or detracted from, the CFGB completing its work and reaching its objectives? (e.g. process, structure, resources, external influences/barriers) (30 minutes)

2. Are there more efficient and/or effective ways of achieving the CFGB’s objectives? (30 minutes)

Are there ways to improve the CFGB’s processes?
Are there ways of improving the quality of the CFGB’s work?
Are there other approaches/alternative delivery models that would be more efficient or effective?

3. What are the most important best practices and lessons learned from CFGB’s experience? (e.g. reviewing cases, issuing F&Rs, identifying systemic issues, communicating with the CDS and CF members)? (20 minutes)

Footnotes

Footnote 1

Adapted from the Chief of the Defence Staff Annual Report on The Canadian Forces Grievance System. January – December 2008. p4-5

Return to footnote 1 referrer

Footnote 2

Review of the Canadian Forces Grievance Board Internal Case Management Processes, John Frecker, Legistec Inc., 1 May 2008; p.2

Return to footnote 2 referrer

Footnote 3

Ibid.

Return to footnote 3 referrer

Footnote 4

Canada First Defence Strategy p.16

Return to footnote 4 referrer

Footnote 5

Canada First Defence Strategy p.16

Return to footnote 5 referrer

Footnote 6

CFGB 2005 Annual report, p8

Return to footnote 6 referrer

Footnote 7

The First Independent Review by the Right Honourable Antonio Lamer P.C., C.C., C.D. 2003 p 87

Return to footnote 7 referrer

Footnote 8

Treasury Board of Canada Secretariat. (2009). Policy on Evaluation, April 1, 2009. Accessed online 24/05/10 at http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024&section=text#cha4

Return to footnote 8 referrer

Footnote 9

The Principled Approach would deviate from the process described in 1.2 in that all grievances at the FA level would first be analyzed by the CFGA, regardless of subject matter. If the CFGA decided to grant the grievance, the process would end there. However, if the grievance is denied, the file would be transferred to the CFGB for review. A change to the NDA and the QR&O is required to implement this approach.

Return to footnote 9 referrer

Evaluation of the CFGB Review of Military Grievances Program

Table of Contents

List of Acronyms

Executive Summary

Description of the Program

Methodology

Relevance

Performance (Effectiveness)

Performance (Efficiency & Economy)

1. Introduction

1.1 Program Background

1.2 Overview of the Canadian Forces Grievance SystemFootnote 1

1.3 Profile of the CFGB Review of Military Grievances Program

2. Evaluation Issues and Methodology

2.1 Evaluation Issues and Questions

Relevance

Performance (effectiveness, efficiency and economy)

2.2 Data Collection Methods

2.2.1 Document Review

2.2.2 Interviews

2.2.3 Focus Group

2.2.4 Data Analysis

2.3 Limitations

Limited External Analysis of the CFGB

Reporting time to Grievances

CF members could not be surveyed

Change of Strategic Outcome

Conclusion

3. Conclusions & Supporting Findings

3.1 Relevance

3.1.1 Is there a continued need for an external review of military grievances?

3.1.2 Do the objectives of the CFGB continue to align with government priorities?

3.1.3 Do the CFGB’s activities align with federal roles and responsibilities?

3.2 Performance (Effectiveness)

3.2.1 Does the CDS have an increased awareness and understanding of systemic issues?

3.2.2 Does the CFGB assist the CDS in rendering decisions on grievances?

3.2.3 Has the work of the CFGB contributed to an increase in stakeholders’ awareness and understanding?

3.2.4 Have regulations, policies, guidelines and the grievance process been improved as a result of the CFGB’s work?

3.2.5 What impact has the work of the CFGB had on confidence levels in the grievance process?

Stakeholders Interviews

CFGB’s Grievors Survey

3.2.6 Have the F&Rs of the CFGB been implemented and led to improvements in CF conditions of service?

3.2.7 What factors contributed to, or detracted from, the CFGB completing its work and reaching its objectives?

3.3 Performance (Efficiency & Economy)

3.3.1 Are there more effective and efficient means of achieving the CFGB’s objectives?

3.3.2 Is the CFGB delivering this program efficiently? How could this be improved?

4. Summary and Recommendations

4.1 Relevance

4.2 Performance (Effectiveness)

4.3 Performance (Efficiency & Economy)

Appendix A: Logic Model

Appendix B: Evaluation Matrix

Appendix C: List of Documents Reviewed

Appendix D: Interview Guides

Interview Guide – CDS & DGCFGA

Interview Guide – VCDS, CMP & IAs

Interview Guide – Counsel of Grievors

Interview Guide – Internal Stakeholders

Appendix E: Focus Group Guide

Footnotes

1.2 Overview of the Canadian Forces Grievance System^{Footnote 1}