Assessment of physician performance in Alberta: the Physician Achievement Review

William Hall,*† MD; Claudio Violato,‡ PhD; Raymond Lewkonia,*† MB; Jocelyn Lockyer,* MHA; Herta Fidler,* MSc; John  Toews,* MD; Penny Jennett,*† PhD; Michel Donoff,†§ MD; David Moores,†§ MD

CMAJ 1999;161:52-7.

• Abstract
• Introduction
• Development of a program to assess physician performance
• Methods for pilot feasibility study
• Results of the pilot study
• Implementation of the PAR program
• Conclusion
• References

The College of Physicians and Surgeons of Alberta, in collaboration with the Universities of Calgary and Alberta, has developed a program to routinely assess the performance of physicians, intended primarily for quality improvement in medical practice. The Physician Achievement Review (PAR) provides a multidimensional view of performance through structured feedback to physicians. The program will also provide a new mechanism for identifying physicians for whom more detailed assessment of practice performance or medical competence may be needed. Questionnaires were created to assess an array of performance attributes, and then appropriate assessors were designated — the physician himself or herself (self-evaluation), patients, medical peers, consultants and referring physicians, and nonphysician coworkers. A pilot study with 308 physician volunteers was used to evaluate the psychometric and statistical properties of the questionnaires and to develop operating policies. The pilot surveys showed good statistical validity and technical reliability of the PAR questionnaires. For only 28 (9.1%) of the physicians were the PAR results more than one standard deviation from the peer group means for 3 or more of the 5 major domains of assessment (self, patients, peers, consultants and coworkers). In post-survey feedback, two-thirds of the physicians indicated that they were considering or had implemented changes to their medical practice on the basis of their PAR data. The estimated operating cost of the PAR program is approximately $200 per physician. In February 1999, on the basis of the operating experience and the results of the pilot survey, the College of Physicians and Surgeons of Alberta implemented this innovative program, in which all Alberta physicians will be required to participate every 5 years.

[Contents]

The provincial medical licensing authorities have established procedures to assess the small minority of physicians who may have problems with loss of competence or unacceptable professional conduct or against whom allegations of deficient performance have been made.^1,2,3 These programs are expensive, and their focus on questionable performance does not assist or assure the large majority of physicians without such problems.

In 1995 the College of Physicians and Surgeons of Alberta established the Physician Performance Advisory Committee to establish a process to evaluate physician performance; members were drawn from the 2 Alberta medical schools (the Universities of Alberta and Calgary) and the Alberta Medical Association and included a representative of the public.

In 1998 more than half of the 770 complaints to the College of Physicians and Surgeons of Alberta related to issues of physician­patient communication (Donald E. Chadsey, Deputy Registrar: personal communication, 1999). The Physician Performance Advisory Committee anticipated that a process drawing to the attention of physicians important issues such as physician­patient communication might improve the quality of medical practice and avert many such complaints. There is evidence that this strategy can be effective. For example, the peer review program of the College of Physicians and Surgeons of Ontario showed that the office records of randomly selected physicians tended to improve after defects were pointed out.⁴ A British study suggested that the routine practice of doctors can be influenced by feedback from patients.⁵ Other studies have reported the feasibility and value of physician performance appraisal by patients,^6,7,8,9 peers^10,11 and hospital nurses.^12,13

This paper describes the purpose, development and pilot studies of a program that will regularly assess the performance of all licensed physicians in Alberta. The process may trigger further assessment of a limited number of physicians.

[Contents]

After its initial work, the Physician Performance Advisory Committee concluded that a performance assessment program should conform to some specific concepts and characteristics (Table 1). The committee selected the program name Physician Achievement Review (PAR) to denote a supportive purpose and the goals of describing professional accomplishment and improving practice.

Extensive discussions within the committee generated 2 complementary inventories of medical practice characteristics: first, 6 broad categories of physician performance attributes — medical knowledge and skills, attitudes and behaviour, professional responsibilities, practice improvement activities, administrative skills and personal health — and second, 5 categories of sources of physician assessment — self, patients, medical peers or colleagues, consulting physicians to whom patients are referred and nonphysician coworkers in health care (e.g., secretaries, nurses and pharmacists). The Physician Performance Advisory Committee used these inventories to construct an assessment grid that listed 90 performance attributes and appropriate raters. The committee then classified the attributes in relation to the members' judgement of good medical care as essential, important, desirable or irrelevant. The performance grid categories were used to develop separate questionnaire instruments reflecting the perspectives of self, patients, medical peers, consultants and nonphysician coworkers respectively (Table 2).

Pilot projects

The College of Physicians and Surgeons of Alberta contracted a group from the University of Calgary (Faculty of Medicine — Continuing Medical Education and Faculty of Education) to further refine the questionnaires through a prepilot survey in 1996 with 28 volunteer physician participants¹⁵ and a larger pilot feasibility study in 1997 with 308 participants, the results of which are reported here. The results of the pilot study form the basis of the evaluation tools now being used in Alberta. Both studies were approved by the University of Calgary Research Ethics Committee. In the prepilot study, the coefficient of generalizability (Cronbach alpha) ranged from 0.64 with 4 peer raters to 0.82 with 10 peer raters, which provided satisfactory technical evidence of internal consistency.

The questionnaires developed from the performance grid were further evaluated by focus groups of patients, generalist and specialist physicians, and other health care workers (mostly nurses) and were adapted for the perspectives of the assessors — self, patients, peers, consultants and coworkers.

[Contents]

The goals of the pilot study were to evaluate the psychometric properties and the statistical validity of the questionnaires, to compare self-ratings with ratings given by peers and co-workers, and to examine the influence of raters' familiarity with the physicians being assessed. A sample of approximately 300 physicians was required; 874 physicians were contacted to meet this sample size. The inclusion criteria were registration with the College of Physicians and Surgeons of Alberta for more than 5 years and current practice as a family physician, general practitioner, or specialist in obstetrics and gynecology, general internal medicine, or general pediatrics. Participants were recruited by means of a letter sent by the registrar of the College of Physicians and Surgeons of Alberta to a stratified random sample of physicians.

Questionnaire instruments

Participants were asked to complete a self-assessment questionnaire and to identify patients, physicians and other health care coworkers to complete the other assessment questionnaires. A previous study¹¹ reported that the method of selecting peer or patient raters (by the physician, by the investigators, or at random from lists of associates or patients) does not influence ratings. Accordingly, we asked participants to identify 6 peers, 6 physicians to whom they referred patients, and 6 nonphysician coworkers to complete the respective assessment questionnaires. In addition, consulting physicians were asked to identify 6 referring physicians. To select patient assessors, the participants were asked to have their office staff distribute the questionnaires systematically (e.g., every second patient beginning Monday morning). All of the raters were asked to rate statements on a 5-point Likert scale. Each statement began with the phrase "Compared to the physicians I know, this one is ...," where 1 = among the worst, 2 = bottom half, 3 = average, 4 = top half, and 5 = among the best ("unable to assess" was also an option). Raters were also asked how well they knew the physician.

Statistical methods

The analyses focused on descriptive statistics and reliability (as indicated by the Cronbach alpha statistic). Differences in ratings between self-assessments and the assessments of peers and coworkers were evaluated by Student's t-test. The potential influence of self-reported familiarity with the physician was tested by one-way multivariate analyses of variance (MANOVA).

[Contents]

Descriptive statistics

Most (221 or 71.8%) of the 308 volunteer physicians were men, 192 (62.3%) had received their medical degree in Canada, 224 (72.7%) had been in practice between 10 and 30 years, 108 (35.1%) were in rural practice and 51 (16.6%) were specialists. The mean ratings on all of the questionnaires were greater than 3.5 (Table 3). The Cronbach alpha coefficients were high (above 0.90), which provided evidence for the internal consistency and technical reliability of all of the questionnaires. On the questionnaires used by patients, peers, consulting or referring physicians, and nonphysician coworkers, the options "unable to assess" and "not applicable" were selected infrequently (in less than 10% of responses). For 28 physicians (9.1%) the PAR results were more than one standard deviation from the mean for their peer group for 3 or more of the 5 major domains of assessment (self, patients, peers, consultants and coworkers).

Influence of rater familiarity with physician

Knowledge of the physician was used as an independent variable in a one-way MANOVA with the questionnaire items as dependent measures. For the peer assessment questionnaire, there was an overall main effect (Wilk's lambda = 0.88, F = 21.11, p < 0.001), which indicates that the results were affected by the peers' knowledge of the physician. There were statistically significant differences for almost half of the items, and for most of these, the raters who indicated that they knew the physicians not well or not at all gave more favourable ratings than raters who knew the physicians somewhat, well or very well. Raters who did not know the physicians well constituted only a small percentage of all raters (1.8%), and the mean difference in ratings was also small (0.21). The MANOVA results also showed overall significant effects for the coworker questionnaire (Wilk's lambda = 0.88, F = 11.58, p < 0.001). As for the peer assessment questionnaire, it was the raters who indicated that they knew the physician not well or not at all who gave the highest ratings. The differences in ratings were small, and only 1.3% of raters gave high ratings. Comparison of self-assessments with assessments by peers, consultant and referring physicians, and coworkers showed little correspondence between the 2 categories. Physicians rated themselves less favourably than did their professional colleagues. Some of the consultants said that they did not know enough about their referring physicians to appropriately rate performance. Therefore, for implementation of the PAR program it was decided to merge the peer and consultant questionnaires into a single questionnaire for medical colleagues.

The volunteer physicians were asked, in focus groups and by questionnaires, about their reactions to their PAR profiles. Approximately two-thirds of respondents reported having contemplated or initiated changes in practice as a result of their PAR profiles.¹⁶ The most commonly mentioned changes related to communication with patients.

[Contents]

On the basis of a recommendation from the Physician Performance Advisory Committee, the Council of the College of Physicians and Surgeons of Alberta implemented the PAR program for all physicians in February 1999. Some practical aspects of the evolving PAR program are described here.

There are 4700 physicians in Alberta, and it is planned that approximately 20% will be assessed each year over a 5-year cycle. For the assessment, a PAR package containing an explanation of the process and questionnaires will be mailed to each physician. The physician will complete the self-evaluation questionnaire (consisting of 26 questions) and will distribute questionnaires to the following assessors: 25 patients (44 questions), 8 medical colleagues (peers or consultants; 26 questions) and 6 nonphysician health care coworkers (17 questions). The completed questionnaires will be returned to the data processing centre in prepaid envelopes.

The PAR responses will be sent to participating physicians in text and graphic formats. Each physician's performance report will be returned only to that individual and the Physician Performance Committee. Performance data will be classified according to the domain of assessment (as in Table 2) and in relation to reference ranges for peer groups. A guide to interpreting the results will be provided. The PAR report will include the names and phone numbers of contact persons for further discussion about the interpretation of PAR profiles, as well as a list of educational resources.

The confidentiality of individual physician data will be assured by amendments to the Medical Professions Act of Alberta, which were enacted in 1998.¹⁷ The legislation states that a physician shall not be required to participate in the PAR program more than once every 5 years.

Review of the PAR results will be the responsibility of a committee (Physician Performance Committee) accountable to the Council of the College of Physicians and Surgeons of Alberta. This committee will examine analyses of the results and, on the basis of operating experience, will consider modifications to PAR procedures, such as changes in the number and content of questionnaires. This committee will evaluate the value and outcomes of the PAR program. Program monitoring will cover the educational activities reported by physicians in their annual license reapplication to the college, the profile of patients' complaints to the college, and practice changes observed or reported in research studies.

Participation in the PAR program will be required unless a physician can explain to the Physician Performance Committee why some or all of the process is inappropriate for his or her practice. Refusal to participate without reasonable grounds may be regarded as conduct subject to the disciplinary processes of the College of Physicians and Surgeons of Alberta.

The PAR has been developed with the majority of Alberta physicians in mind: those who engage in office and hospital practice with substantial direct patient care. Specialty groups, particularly procedural and diagnostic specialties, have been invited to assist in developing modified questionnaires to assess their performance.

The administrative costs of the PAR program will be approximately $200 per physician (in 1999 dollars). The PAR program will be funded through the operating budget of the College of Physicians and Surgeons of Alberta. The necessary increase in annual licence fees of $40 per year for all Alberta physicians is regarded as a cost inherent to the self-regulation function of the profession.

The primary objective of the PAR will be improvement of practice quality. This is the anticipated result of informing physicians of possible performance deficiencies, as in the model of the quality improvement cycle¹⁴ and the educational model of practice reflection.¹⁸ Education related to the PAR will be voluntary for most physicians. If serious performance deficiencies are identified during review by the Physician Performance Committee, the College of Physicians and Surgeons of Alberta, under its existing authority, may require and direct detailed evaluation, remedial education and subsequent reassessment by methods appropriate to the deficiencies.

The PAR office will maintain an inventory of educational resources for professional and nonprofessional help, for example relating to physician­patient communication or practice management. This inventory will be available to all Alberta physicians, not only those participating in the PAR program in a particular year.

The PAR process will not replace existing procedures by which the College of Physicians and Surgeons of Alberta fulfills its statutory obligations under the Medical Profession Act of Alberta, such as investigation of complaints. Now that the program has been implemented, the threshold criteria for considering further assessment will be determined by the Physician Performance Committee from operating experience and its review of aggregate performance data. The results of the PAR pilot study indicate that for approximately 10% of physicians, further assessment or directed education may be appropriate.

[Contents]

The PAR program has been developed by a partnership of the College of Physicians and Surgeons of Alberta, the 2 Alberta medical schools and members of the medical profession, including 336 physicians who participated in 2 pilot projects.

The program has been designed to be relevant to aspects of medical practice judged to be important for most physicians in Alberta. Most of the reported research on assessing physician performance by questionnaires has used the ratings of single groups of raters, such as patients, peers or coworkers. Our findings suggest that there are practical and technical advantages to the multidimensional perspective provided by the different groups.

The PAR program aims to assist the physicians of Alberta to identify aspects of their circumstances and practice that they might wish to review and perhaps change or improve, for the benefit of their patients and their own professional satisfaction.

Further information about the College of Physicians and Surgeons of Alberta process is available at a Web site under development (www.par-program.org). The intent of this site is to provide information to physicians regarding pertinent continuing medical education.

Dr. Anthony Marini assisted with the psychometric analyses. Members of the Physician Performance Advisory Committee, in addition to the authors, were Lynn Malin (member of the lay public) and Drs. Gordon Arnett, Keith Brownell, Betty Flagler, Susan Low, Wolfgang Schneider and Brynn Whittaker. The committee's work was ably supported by Dr. Bryan Ward and John Swiniarski, assistant registrars of the College of Physicians and Surgeons of Alberta. We thank the registrar of the college, Dr. Larry Ohlhauser, for his support throughout this development project, which was funded by the college.

Competing interests: None declared.

[Contents]

Send a letter to the editor Envoyez une lettre à la rédaction

Reprint requests to: Dr. William G. Hall, Health Sciences Centre, University of Calgary, 3030 Hospital Dr. NW, Calgary AB T2N 4N1; fax 403 670-1240

1. Page GG, Bates J, Dyer SM, Vincent DR, Bordage G, Jacques A, et al. Physician-assessment and physician-enhancement programs in Canada. CMAJ 1995;153:1723-8. [MEDLINE]
2. Cunnington JP, Hanna E, Turnbull J, Kaigas TE, Nornan GR. Defensible assessment of the competency of the practicing physician. Acad Med 1997;72:9-12. [MEDLINE]
3. McAuley RG, Paul WM, Morrison GH, Beckett RF, Goldsmith CH. Five-year results of the peer assessment program of the College of Physicians and Surgeons of Ontario. CMAJ 1990;143:1193-9. [MEDLINE]
4. Norton PG, Dunn EV, Beckett R, Faulkner D. Long term follow-up in the Peer Assessment Program for non-specialist physicians in Ontario, Canada. Jt Comm J Qual Improv 1998;24:334-41. [MEDLINE]
5. Hearnshaw H, Baker R, Cooper A, Eccles M, Soper S. The costs and benefits of asking patients for their opinions about general practice. Fam Pract 1996;13:52-8. [MEDLINE]
6. Lewis JR, Williamson V. Examining patient perceptions of quality care in general practice: comparison of quantitative and qualitative methods. Br J Gen Pract 1995;45:249-53. [MEDLINE]
7. Grogan S, Conner M, Willits D, Norman P. Development of a questionnaire to measure patients' satisfaction with general practitioners' services. Br J Gen Pract 1995;45:525-9. [MEDLINE]
8. Carey KG, Seibert JR. A patient survey system to measure quality improvement: questionnaire reliability and validity. Med Care 1993;31:834-45. [MEDLINE]
9. Weaver MH, Ow CL, Walker DJ, Degenhardt EF. A questionnaire for patients' evaluations of their physicians' humanistic qualities. J Gen Intern Med 1993;8:135-9. [MEDLINE]
10. Ramsey PG, Wenrich MD, Carline JD, Inui TS, Larson EB, LoGerfo JP. Use of peer ratings to evaluate physician performance. JAMA 1993;269:1655-60. [MEDLINE]
11. Ramsey PG, Carline JD, Blank LL, Wenrich MD. Feasibility of hospital-based use of peer ratings to evaluate the performances of practicing physicians. Acad Med 1996;71:364-70. [MEDLINE]
12. Wenrich MD, Carline JD, Giles LM, Ramsey PG. Ratings of the performances of practicing internists by hospital-based registered nurses. Acad Med 1993;68:680-7. [MEDLINE]
13. Wooliscroft JO, Howell JD, Patel BP, Swanson DB. Resident­patient interaction: the humanistic qualities of internal medicine residents assessed by patients, attending physicians, program supervisors and nurses. Acad Med 1994;69:216-24. [MEDLINE]
14. Berwick DM. Continuous improvement as an ideal in health care. N Engl J Med 1989;320:53-6. [MEDLINE]
15. Violato C, Marini A, Toews J, Lockyer J, Fidler H. Using peers, self, patients and co-workers to assess physician performance. Acad Med 1997;72:582-4.
16. Fidler H, Lockyer J, Toews J, Violato C. Changing physician practice. The effect of individual feedback. Acad Med 1999;74(6):77-89.
17. Medical Professions Act, RSA 1980, cM-12, as am. by SA 1988 c21.
18. Schon DA. Educating the reflective practitioner. San Francisco: Josey-Bass; 1987.

© 1999 Canadian Medical Association (text and abstract)