Does the site of postgraduate family medicine training predict performance on summative examinations? A comparison of urban and remote programs

Robert J. McKendry, Nick Busing, Dale W. Dauphinee, Carlos A. Brailovsky, André-Philippe Boulais

CMAJ 2000;163(6):708-11

See:  eLetters

[PDF]

Background: The location of postgraduate medical training is shifting from teaching hospitals in urban centres to community practice in rural and remote settings. We were interested in knowing whether learning, as measured by summative examinations, was comparable between graduates who trained in urban centres and those who trained in remote and rural settings.

Methods: Family medicine training programs in Ontario were selected as a model of postgraduate medical training. The results of the 2 summative examinations — the Medical Council of Canada Qualifying Examination (MCCQE) Part II and the College of Family Physicians of Canada (CFPC) certification examination — for graduates of the programs at Ontario's 5 medical schools were compared with the results for graduates of the programs in Sudbury and Thunder Bay from 1994 to 1997. The comparability of these 2 cohorts at entry into training was evaluated using the results of their MCCQE Part I, completed just before the family medicine training.

Results: Between 1994 and 1997, 1013 graduates of family medicine programs (922 at the medical schools and 91 at the remote sites) completed the CFPC certification examination; a subset of 663 completed both the MCCQE Part I and the MCCQE Part II. The MCCQE Part I results for graduates in the remote programs did not differ significantly from those for graduates entering the programs in the medical schools (mean score 531.3 [standard deviation (SD) 69.8] and 521.8 [SD 74.4] respectively, p = 0.33). The MCCQE Part II results did not differ significantly between the 2 groups either (mean score 555.1 [SD 71.7] and 545.0 [SD 76.4] respectively, p = 0.32). Similarly, there were no consistent, significant differences in the results of the CFPC certification examination between the 2 groups.

Interpretation: In this model of postgraduate medical training, learning was comparable between trainees in urban family medicine programs and those in rural, community-based programs. The reasons why this outcome might be unexpected and the limitations on the generalizability of these results are discussed.

[Contents]

Using these family medicine training programs as a model for postgraduate medical education, we were interested in comparing the performance of trainees in the traditional programs based in medical schools with the performance of those in the remote programs. Performance was measured using standardized, validated summative examinations: the Medical Council of Canada Qualifying Examination (MCCQE) Part II and the College of Family Physicians of Canada (CFPC) certification examination.

[Contents]

Through the Canadian Resident Matching Service (CaRMS) applicants for Ontario family medicine training programs must specifically request remote training in either Sudbury or Thunder Bay or training at 1 of the province's 5 medical schools. About 12 candidates are selected for each remote program per year. Before entering postgraduate training, all Canadian medical graduates take the MCCQE Part I. This exam consists of 3 multiple-choice components and a fourth component comprising a short menu and short-answer questions. We used the MCCQE Part I results to determine whether the examination performance was comparable between graduates entering the remote programs and those entering the urban programs.

Toward the end of the 2 years of family medicine training, most graduates complete the MCCQE Part II and the CFPC certification examination. The MCCQE Part II, an Objective Structured Clinical Examination (OSCE), is usually written 8 months before completion of the training program. The 2-part CFPC certification examination, comprising written short-answer management problems and 5 simulated office oral examinations, is undertaken 2 months before the end of the training program and is intended to evaluate the competencies required for family practice in Canada.⁴ A 1992 medical school graduate, for example, would usually take these 3 examinations over 3 consecutive years. The format and performance characteristics of the CFPC certification examination and the MCCQE Parts I and II have been well reviewed.^5,6,7,8,9,10

We obtained the results of the MCCQE Part II and the CFPC certification examination for all trainees in the 5 school-based urban programs and the 2 remote programs from 1993 to 1997. Because the format of the CFPC examination was in transition in 1993, we included only the results for 1994 to 1997 in our comparison. Only aggregate data were provided, to protect the anonymity of individual results. The Ottawa Hospital Research Ethics Board approved the study protocol.

For the results of the CFPC certification examination, 2 types of statistical analyses were performed. First, descriptive statistics were performed for the short-answer management problems and the simulated office oral examinations, followed by a factorial analysis of variance (ANOVA) to study differences between group means. For all the studies, homogeneity of group variances was estimated using Levene's test.¹¹ Second, generalizability studies were conducted using the EtudGen program developed by McNicoll and associates¹² to establish whether training sites have an impact on candidates' scores in a similar manner as already described.¹³ The design used for this purpose was residents nested in training sites by cases (R:S × C).

MCCQE results were identified only for graduates who had taken both the MCCQE Part I and the MCCQE Part II. The aggregate scores for these individuals were provided by the evaluation bureau of the Medical Council of Canada. Because 1992 was a transition year for the MCCQE Part II in moving to the current OSCE format, only the results for 1993 to 1997 were considered in our analysis. A 2-tailed test was chosen to test the null hypothesis that mean examination scores would be equal for both Parts I and II of the MCCQE. A multivariate ANOVA was performed using site of residency training as an independent variable and MCCQE Part I and Part II results as dependent variables.

[Contents]

From 1994 to 1997 a total of 1013 family medicine trainees (922 in the school-based urban programs and 91 in the remote programs) completed the CFPC certification examination. The mean scores are presented in Table 1. Scores for the 2 components of the examination were compared separately, and the combined mean scores for the 2 remote programs were compared with the combined mean scores for the 5 urban programs. As well, the mean scores for graduates from each remote program were compared with the mean scores for graduates from its parent medical school programs. There was no trend toward consistently higher marks in any of the groups. Levene's test for homogeneity of variance indicated that the different group variances were equal, allowing interpretation of the ANOVA results.

The generalizability studies showed that the variance for residents nested in sites (i.e., the equivalent of the true variance) represented about 10% of the variance across the years and the components of the CFPC examination. The variance for training sites represented less than 1.5% of the total variance. The variance for item difficulty related to the training site was represented by the interaction between items and sites. It explained less than 1% of the total variance, thus indicating that residents interact with the exam items in the same way, independently of their training program.^12,13

Of the 1013 trainees who completed the CFPC certification examination, 663 completed both Parts I and II of the MCCQE. (The number of trainees who took the CFPC examination differs from the number who took the MCCQE Part II because only physicians who graduated medical school before Dec. 31, 1999, had to obtain a passing score on the MCCQE Part II.) In terms of performance before and after training in family medicine, the MCCQE Part I and Part II results for those in the remote programs did not differ significantly from those for graduates in the urban programs (Table 2).

[Contents]

Conducting postgraduate medical training entirely in a community setting is promoted as a way to better understand and meet that community's health care needs. Training in rural and remote communities is one example. As a model system, we compared the examination performance of family medicine residents in remote training programs in northern Ontario, hundreds of kilometres from any medical school, with the performance of residents in traditional urban training programs in southern Ontario. A difference in performance might be expected for many reasons:

• Candidate characteristics: For example, compared with candidates who choose the traditional urban programs, are those who choose the new, more remote training programs more adventurous and self-sufficient people with superior self-learning skills?
• Program characteristics: Would the remote programs have "growing pains" in the initial start-up period? Would the faculty in the remote programs, recruited from existing community physicians, provide similar quality of training to that offered in the larger, urban programs? Would the supplemental grants provided to the remote programs give them a resource advantage?
• Relevance of examination measures: Are the items measured by the examinations equally relevant to urban and remote family practice?

The results of our study supported our hypothesis that there would be no difference in the examination performance between graduates of the remote programs and those of the urban programs. One might ask why there was no consistent difference despite various prevailing conditions that hypothetically could influence the examination performance of one or the other group of graduates. Perhaps the similar examination results were due to careful quality control in the planning of the remote and urban programs, reinforced by periodic accreditation procedures mandated by the CFPC. Could one expect to obtain similar results with specialty and subspecialty training programs at remote sites? How much learning is directly attributable to the teaching and training provided by the programs? If postgraduate training at remote sites is not inferior to that provided at medical schools, what role does the medical school play in remote training?

Competing interests: None declared.

Contributors: Dr. McKendry originated the research questions and study design, coordinated the data acquisition and was primarily responsible for manuscript development. Dr. Busing obtained the CFPC examination results and contributed to the development of the study design and the writing of the paper. Dr. Dauphinee obtained and explained the MCCQE Part I and Part II results and contributed to the development of the study design and the writing of the paper. Dr. Brailovsky and Mr. Boulais performed the statistical analysis of the CFPC and MCCQE results and reviewed the statistics component of the paper.

[Contents]

Reprint requests to: Dr. Robert J. McKendry, Department of Medicine, Ottawa Hospital — General Campus, 501 Smyth Rd., Ottawa ON K1H 8L6

1.	Rabinowitz HK. Recruitment, retention, and follow-up of graduates of a program to increase the number of family physicians in rural and underserviced areas. N Engl J Med 1993;328:934-9. [MEDLINE]
2.	Brazeau NK, Potts MJ, Hickner JM. The Upper Peninsula Program: a successful model for increasing primary care physicians in rural areas. Fam Med 1990;22:350-5. [MEDLINE]
3.	Rosenblatt RA, Whitcomb ME, Cullen TJ, Lishner DM, Hart LG. Which medical schools produce rural physicians? JAMA 1992;268:1559-65. [MEDLINE]
4.	College of Family Physicians of Canada. Canadian family medicine: education objectives for certification of family medicine. Willowdale (ON): The College; 1974.
5.	Dauphinee WD, Baumber JS, Reznick RK, Berard MJ, Blackmore DE. Rationalizing a licensing examination system: the new examinations of the Medical Council of Canada. In: Proceedings of the Sixth Ottawa Conference on Medical Education; 1994 June 26-29; Toronto. Toronto: University of Toronto; 1994. p. 26-8.
6.	Reznick RK, Blackmore D, Dauphinee WD, Rothman AI, Smee S. Large-scale high-stakes testing with an OSCE: report from the Medical Council of Canada. Acad Med 1997;71(1):S19-21.
7.	Handfield-Jones R, Rainsberry P. Certification examination of the College of Family Physicians of Canada. Part 1: History and implications for the present and future. Can Fam Physician 1996;42:957-69. [MEDLINE]
8.	Handfield-Jones R, Brown JB, Rainsberry P, Brailovsky CA. Certification examination of the College of Family Physicians of Canada. Part 2: Conduct and general performance. Can Fam Physician 1996;42:1188-95. [MEDLINE]
9.	Handfield-Jones R, Brown JB, Biehn J, Rainsberry P, Brailovsky CA. Certification examination of the College of Family Physicians of Canada. Part 3: Short-answer management problems. Can Fam Physician 1996;42:1353-61. [MEDLINE]
10.	Brown JB, Handfield-Jones R, Rainsberry P, Brailovsky CA. Certification examination of the College of Family Physicians of Canada. Part 4: Simulated office orals. Can Fam Physician 1996;42:1539-48. [MEDLINE]
11.	Levene H. Robust tests for the equality of variance. In: Olkin I, editor. Contributions to probability and statistics. Palo Alto (CA): Stanford University Press; 1960. p. 278.
12.	McNicoll A, Brailovsky CA, Bertrand R, Cardinet J. EtudGen, programme pour l'analyse de la généralisabilité pour Macintosh. In: Bain D, Pini G. Pour évaluer vos évaluations: La généralisabilité, mode d'emploi. Geneva: Centre de recherches psychopédagogiques; 1996.
13.	Brailovsky CA, Grand'Maison P, Lescop J. A large-scale multicenter objective structured clinical examination for licensure. Acad Med 1992;67:S37-9. [MEDLINE]

© 2000 Canadian Medical Association or its licensors