 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
| |
|
||||||||||||||||
| |
|
||||||||||||||||
|
|
 The role of lay panelists on grant review panels Abstract The purpose of this study was to investigate the attitudes of scientists and lay people participating in National Cancer Institute of Canada (NCIC) grant review panels towards the inclusion of non-scientists in the review process. Questionnaires were sent to the 126 scientists and 24 lay panelists who participated in NCIC's grant reviews in 1998. Survey topics included lay member selection, the role of the lay panelist and suggestions for improving the process. Data were analyzed qualitatively, and quantitatively using SPSS. Sixty-one of the 126 scientists (48.4%) and 16 of the 24 lay panelists (66.7%) completed the survey. Female scientists were significantly more supportive than male scientists of the selection of cancer patients/survivors/advocates as lay members (p = 0.01), but overall their responses were more similar to those of their male colleagues than of the lay respondents. There were significant differences between the lay and scientist respondents on lay member responsibilities (p = 0.01), the format of lay grant review (p = 0.04), lay member contribution to panel discussion (p = 0.01), and understanding of the lay role (p = 0.02). Key words: Canada; cancer; consumer advocacy; patient advocacy; peer review Introduction When Sharon Batt, Canadian breast cancer survivor, wrote of her experiences with the scientific, health care, and political establishment in Patient No More: The Politics of Breast Cancer,1 she symbolized a growing awakening among cancer patients and survivors to the importance of becoming politically conscious about their illnesses. At the foundation of this “consciousness-raising” phenomenon was a conviction that scientists and the cancer patient community were fundamentally at odds with each other in their approaches to cancer research and treatment. Specifically, as Batt put it, “cancer researchers privilege the extension of the patient's life above all other priorities, regardless of the discomfort of the treatments entailed, while the [breast] cancer patient seeks ways of enhancing quality of life, investigating preventive therapies, and limiting the harshness of painful treatments.”1 At no other time in recent memory has the community of patients been so effective in making their unique voice heard. The circumstances converging to produce this trend are complex and multivariate, but no doubt relate to the demise of reliance on “experts” over the last century, defiance of the previously unquestioned wisdom of traditional authority figures, and the role of the media in debunking previously sacrosanct ideas of authority and expert opinion. The movement at bottom arises from the optimistic ideal that ordinary people can truly influence change in what might sometimes seem to be monolithic bureaucratic communities of government and science. The aspirations of cancer advocates were first embodied in the U.S. Army's Breast Cancer Research Program, which since 1993 has included “consumers” (e.g., breast cancer survivors) in the grant review process. Since 1995, these consumers have participated as full voting members and colleagues of the scientists in the peer review system. The U.S. Army program defines “consumer advocates” as those who have “first-hand experience with breast cancer as affected persons and are active in breast cancer advocacy organizations.”2 The inclusion of such advocates in the process is intended to broaden discussion beyond hard-core science issues to encompass the patient/survivor perspective. The Canadian system has responded similarly. Emblematic of this trend has been the coming together of the Canadian Breast Cancer Research Initiative (CBCRI), begun in 1993 as a partnership among the foremost Canadian research bodies, the federal government, and the major breast cancer advocacy groups. The CBCRI is the primary impetus behind the funding of breast cancer research in this country. Its inception stemmed in large part from the advocacy efforts of breast cancer survivors and women in Parliament, who insisted that greater efforts be undertaken to fight the disease. These calls for action culminated in the National Forum on Breast Cancer, held in Montreal in 1993, where a consensus emerged that a centralized initiative in funding was needed to sustain research into a cure for the disease. At the core of this consensus was the notion that government and the scientific community must be held accountable for their decisions, and that the ordinary citizen with a vested interest in these decisions must be given a voice. The role of the consumer, then, was elevated as a vehicle through which the scientific and governmental communities could be made more accountable to the public at large and discussion at the table of scientific and governmental decision-making could be injected with a unique societal, humanistic voice. Since 1997, the National Cancer Institute of Canada (NCIC), traditionally the most significant funding organization in the country for cancer research, has made its grant review process more patient-centred by including two non-scientists on each of 12 panels in its grant application review process. The NCIC has drawn from a list of nominees from about 50 organizations to fill its lay panel positions. Resumés and letters written by those nominating the lay candidates have been reviewed by a small panel, and then panelists have been chosen randomly from those candidates approved. Since the lay “experiment” began, the NCIC has undertaken informal assessments of the climate in which the lay panelists were received by the scientists. While “no negative comment has been heard by the NCIC staff from any panel member,”3 this is the first comprehensive survey of either lay or scientist panelists. Objectives A survey of the 24 lay and 126 scientist panelists who participated in the NCIC's grant review process in 1998 was conducted to ascertain the opinions of both groups on the lay panelist role. The study's jumping-off point was simply this: has the scientific establishment recoiled at attempts by the consumer advocacy movement to “share ownership of the research ‘property'” as has been suggested by Batt.1 The present study thus aimed to examine how both groups, lay and scientist panelists, view the legitimacy and significance of the lay role. Given that most of the consumer advocates are women and most of the scientists are men, do gender differences figure into this equation? How similar are the lay panelists' and scientists' characterizations of the interaction they experienced on their respective panels, and what suggestions do they have for improving that interaction? Framing this inquiry was a desire to investigate lay panelists' views on their own preparedness for the process and how orientation sessions could better accommodate their needs in this regard, as well as both groups' views on the criteria that should be involved in the selection of lay panelists. Methods All scientists (126) and lay panelists (24) who had served on the NCIC scientific grant review panels in 1998 were surveyed within four months of their participation. Two separate surveys with overlapping questions were distributed by mail. Each survey consisted of a mixture of closed-ended (multiple choice) and open-ended questions derived from informal feedback already gathered from lay panelists and from discussion with key informants. The surveys were divided into several sections, including lay member selection, orientation and preparation (of each respective group), the lay panelist role, experience on the panel, suggestions for improving the process, and evaluation and feedback. Survey responses were analyzed both qualitatively for themes and illustrative quotes (by both investigators), and quantitatively by using SPSS to determine frequencies and the results of chi square tests. Only themes and quotes agreed to by both authors were selected for inclusion. Results Sixty-one of the 126 scientists (48.4%) and 16 of the 24 lay panelists (66.7%) completed and returned the survey. Of the 16 lay respondents, 13 were women; of the 43 scientists who identified their sex, 9 were women. Lay member selection No significant difference between the lay and scientist panelists was found on the question of lay member selection and background. However, the scientist respondents were more supportive of having cancer patients/survivors sit as lay panelists than the lay respondents themselves (65.0%, n = 26 versus 50.0%, n = 8). This position was reversed with regard to support for having cancer advocates as lay panelists (75%, n = 12, of lay panelists versus 47.6%, n = 19, of scientists) (Table 1). In addition, there were significant sex differences among the scientist respondents on this issue. All the female scientists (n = 9) supported choosing both cancer patients/survivors and relatives of patients/survivors as lay members, as compared with only 54.8% (p = 0.01) and 45.2% (p = 0.003) of their male counterparts respectively (n = 31). The proportion of female scientists supporting cancer advocates (77.8%) was similar to that of the lay respondents, in contrast to the proportion among male scientists (38.7%) (p = 0.04). Lay panelists' roles and responsibilities When asked if it would be beneficial for lay panelists to sit as observers for a year before assuming full responsibilities, 77.0% of scientists agreed, as compared with only 43.8% of lay respondents (p = 0.01). However, all lay panelists felt that some form of orientation session was desirable. A large minority (29.4%) of lay panelists believed that lay review of grant applications should approach that undertaken by the scientists (in the form of an overall numeric rating), as compared with only 4.8% of scientist respondents with this view (p = 0.04). |
|||||||||||||||||||||||||
|
||||||||||||||||||||||||||
|
The lay respondents clearly viewed themselves as fulfilling an ambassadorial role. In fact 94.1% of lay respondents characterized themselves as representatives of the grant review process for their affiliated organizations or the general public. Nevertheless, they expressed ambivalence about their contribution to panel discussion. Although a majority of lay panelists believed that the lay contribution served a worthwhile purpose, only a minority of them (37.5%) characterized it as “significant”. However, clearly a majority of both lay and scientist panelists endorsed the legitimacy of the lay “experiment”. Interaction on the panel There was no significant difference between the male and female scientists on how highly they rated the interaction between lay and scientist panelists. Female scientists had a mean rating of 8.33 (highest score of 10), as compared with a mean of 7.17 by their male counterparts. (There was likewise no significant difference between lay and scientist panelists on this point.) The female scientists responded in ways more similar to their male scientist counterparts than to the lay respondents: 66.7% of female and 67.7% of male scientists believed that lay panelists had “brought a different perspective” to panel discussions. Furthermore, a majority of both male (96.4%) and female (100.0%) scientists believed that an awareness existed on their panels that the lay participants were fulfilling a legitimate role. Both the male and female scientists disagreed that scientists need to better understand the lay role, 72.4% and 77.8% respectively responding “no” to this question. By contrast, there were significant differences between the lay and scientist respondents on these issues. All the lay panelists as compared with 69.8% of scientists (p = 0.01) agreed that lay panelists brought a “different perspective” to the discussions (Table 2). While 91.7% of scientists (n = 55) believed there was an understanding on their panels that lay members were providing a legitimate contribution, only 76.5% of lay respondents (n = 14) believed that such an awareness existed (p = 0.08). Moreover, 62.5% of lay panelists as against only 30.0% of the scientists (p = 0.02) agreed that scientists need to better understand the lay role. Discussion Lay member selection Lay respondents' comments on the selection of lay panel members suggested an emphasis on advocacy as a prerequisite to sitting on the panel. As one lay member put it, lay participants “need some knowledge and experience in the cancer community”; to be “'just interested'” is not enough. For scientist panelists, intelligence and level of education were more important criteria in the selection of lay members. As one scientist stated: “Any interested individual willing to learn about the peer review process and participate in the panel review meetings who has the capacity to undertake this work” should be considered for inclusion on the panel. With respect to scientific expertise, one scientist asserted that lay members “should have some familiarity with the content so they can make meaningful comments.” One of the scientists suggested that cancer patients and survivors possessed an inherent bias that rendered them the most inappropriate candidates for panel participation. This notion contrasted markedly with the lay view that to attempt to acquire scientific expertise would result in the loss of the unique lay viewpoint. Indeed, a clear majority – 75.0% – of the lay respondents indicated they were opposed to the inclusion of a basic science review in their orientation, preferring instead an overview of types of cancer and cancer research (75.0% chose this option), a lay glossary of scientific terms (75.0%), and an introduction to the grant review process itself (93.8%). In the same vein, one female scientist argued that far from being tainted with bias, the lay member serves as a “reality [or] relevance” check on the scientists, as the lay panelist does not harbour an “agenda”. Although lay respondents clearly favoured the idea of cancer advocates sitting as lay panelists, they seemed to support a particular kind of advocacy. Hence, while only 50.0% of lay panelists would choose cancer patients/survivors and 56.3% of them would choose relatives of patients/survivors as lay panelists, 93.8% of lay respondents saw members of cancer organizations as the preferred pool of potential lay members. This finding suggests that the background of these lay respondents may be dominated by an organizational, as opposed to a personal, cancer experience. Their view of advocacy may be correspondingly shaped by this circumstance. In contrast, the women scientists (as distinct from their male counterparts) favoured all varieties of cancer advocates, including patients and survivors, as mentioned previously. Lay member roles and responsibilities Although only a minority of respondents, scientists and lay panelists affirmed that lay panelists had “played a significant role”, most of the lay respondents were not prepared to endorse the suggestion of having lay members serve as mere observers for a year before becoming full-fledged panelists. One put her feelings succinctly when she said that “to go through all the effort that is required to review the grants, then to sit as an ‘observer' would be difficult for me.” Evidently, the “observer” question represented a kind of boundary line that most lay members were unwilling to traverse, given the fundamental legitimacy they saw as embodied in their role. |
||||||||||||||||||||||||||
|
||||||||||||||||||||||||||
Indeed, a majority of the lay respondents believed that they had brought a different perspective from that of the scientists to the panel discussions. All the lay respondents reported that they had provided feedback about their grant review experience to their affiliated organizations, and every one of them believed that this communication had had an impact. One noted that support group members were pleased to hear that professionals were listening to laymen/survivors. A clear majority of lay respondents viewed themselves as ambassadors of the grant review process to the general public. Lay respondents seemed to possess a vision of their role, however tentatively and vaguely defined, as one not only encompassing the relevance of the human perspective (putting “a face on the cancer patient”) within the panel setting but also serving as a conduit between the scientific community and a public that is generally unaware of the inner workings of research and indeed of grant review itself. As at least one scientist remarked, the lay members perpetually emphasized that “science must… be publicly accountable.” Another noted that the lay panelists looked at research proposals with new, untutored eyes, thereby bringing a fresh angle to discussion: “they saw some grants with ‘simple' science to be more valuable than the scientists on the panel.” The scientists were effectively divided among a majority who invested at least some legitimacy in the lay role, and a minority who saw lay members as irrelevant and expendable. Most of the criticism in this regard hinged on the failure of the lay participants to speak the language of science, which most of the scientists viewed as central to the analysis of the grant application. The process, at times, broke down because each group was speaking at cross-purposes with the other. Some scientist respondents disparaged the extra time entailed by the inclusion of lay panelists. As one wrote, “[the lay panelists] took up a great deal of time asking questions … scientists felt obligated to provide lengthy explanations of scientific points that were very difficult for lay members to understand.” How widespread this disaffection with the lay role was among scientists remains unclear, in part because of the difficulties in achieving forthrightness in reporting on questionnaires of this nature. Furthermore, it may be that those respondents with negative views are more readily heard because it is they who arguably relay their feedback most promptly, and express their opinions most fully. Interaction on the panel No significant difference emerged between lay and scientist respondents on the rating of the interaction on their respective panels. Comments of both groups emphasized the pivotal importance of the chairperson of each panel in helping to define the lay role and to structure the form of lay participation. One scientist commented that lay participation was characterized by “poor integration, misunderstanding,” and a sense that the lay members were “‘left out'.” Another alluded to the crucial involvement of the chairperson in setting the tone for lay participation, asserting that at least one chairperson “did not always provide opportunity for inclusion … [of] their concerns.” One lay panelist remarked that during the few instances in which she and her lay counterpart offered comments, the scientists consistently listened to them; she believed that in a few cases the lay contributions changed “the course of events.” Moreover, virtually every one of the lay panelists commented on the new respect and appreciation that the process had inspired in him/her for the work of scientists and the depth of cancer research. The scientists echoed this positive interpretation of rapport on the panel. One scientist described an easy give-and-take interaction, observing that the lay members were encouraged to join any discussion during the panel meeting when they felt they could contribute. Another remarked that the lay panelists he had worked with had performed a “wonderful job”. At the same time, a substantial minority of the lay panelists contended that interaction could be improved by offering the lay members more of a voice on the panel. As one of the lay members noted, the lay panelists would feel that their contribution was more valued if some measure of the relevance of comments offered could be factored into the final ranking decision. Limitations of the study The survey involved a relatively small sample and a response rate slightly below 50% for the scientists. To help formulate the survey questions, NCIC and one of the investigators had interviewed several scientist and lay panelists to ascertain their opinions about the new process. The response rate to the survey may have been reduced if these individuals felt it unnecessary to respond again, in writing. The few female scientist respondents made more detailed analysis difficult. It is questionable, too, given the small sample, how representative of cancer advocates the lay respondents may have been: their emphasis on organizational links (such as with the Canadian Cancer Society) may not be typical of the cancer advocates who sit on the U.S. Army panels, for example. Conclusions The lay members were clearly impressed with the dedication and hard-working nature of the scientists, along with the breadth and depth of cancer research that is being undertaken in Canada. Many of them feel insecure about their contribution, but believe that their confidence will grow with continued experience. They unanimously endorse the legitimacy of lay participation and feel it is an undeniable success. Both theory and practice, they say, must progress, in the sense that the lay role should be better defined and communicated to the scientists and lay members in the hope of improving panel rapport. That definition must be continually shaped by the NCIC, which has taken the lead in Canada in including lay members in the grant review process. The scientist respondents, for their part, represent a heterogeneous group. The extent of their acceptance of lay participation is encouraging; yet, intermingled with this optimism in some cases is doubt about whether lay inclusion is worthwhile. Hence, although a majority of the scientist respondents supported the legitimacy of the lay presence, a minority opposing the lay contribution (at least in its current form) made their voice heard through this survey. The expansion of grant review, which uncharacteristically positions scientists and lay persons together in pursuit of a common goal, has gone some way toward breaking down the barriers between these ‘two solitudes', toward making science more publicly accountable, and toward providing a window onto the world of the scientist. Acknowledgements The authors wish to acknowledge the financial support of the National Cancer Institute of Canada (NCIC), which funded this study. In addition, they wish to thank Dr Robert Phillips, Dr Gareth Taylor, and Dr Marilyn Schneider, all of the NCIC, who contributed to the conceptualization of the study, and offered helpful critical appraisal of the final draft of the article. The authors would also like to acknowledge the support of Jill Cameron, MSc Research Fellow at the University Health Network Women's Health Program, who was involved in the statistical analysis of the survey's results. References
Author References Anne Monahan, Department of Family and Community Medicine, University of Toronto, Ontario, Canada Donna E Stewart, University Health Network and University of Toronto, Ontario, Canada Correspondence: Dr Donna E Stewart, University Health Network, Toronto General Hospital, ML 2-004, 657 University Avenue, Toronto, Ontario, Canada M5G 2N2; Fax: (416) 340-4185; E-mail: donna.stewart@uhn.on.ca [Previous] [Table of Contents] [Next]
|
||||||||||||||||||||||||||
 |
|||
|
Last Updated: 2003-08-25 |  |
|
