Formative Evaluation of Distance Education: A Case Study of the Certificate in Adult Education at the University of Botswana |
Abstract
This paper presents the results of an evaluation of a distance education program at the University of Botswana. This two-year program for the Certificate in Adult Education (CAE) was developed in 1983. The course, intended for adult educators already working in the field, is open to applicants with Junior Certificates (three years of secondary education). During the course, students work with written texts and attend compulsory study weekends every month.
The evaluation presented here focused on two aspects of the program: (1) the administration of distance education and (2) the course content.
The data were gathered through feedback from the monthly study weekends, through group discussion, and from self-administered questionnaires. The survey findings were coded and analyzed using cross tabulations and chi-squares and Pearson's R tests to assess whether apparent differences were significant (at 0.5* and 0.0** levels of significance). A number of variables influenced academic performance, including staying with children,* hours of work per day,** provision of study time by the employer,** availability and use of libraries,* newspaper reading,* and age.*
This paper also looks at the students' perception of the content, value, and quality of the course and the effectiveness of weekend teaching methods. A number of recommendations are made. These include the need for the Institute of Adult Education (IAE) to continue negotiating with employers to provide trainees with reasonable study time during normal working hours, to review and improve methods of submitting monthly assignments, to adapt individual counselling to the trainees' learning needs, and to revise difficult learning units.
Résumé
Cet article présente les conclusions d'une évaluation d'un programme d'enseignement à distance, élaboré en 1983 à l'Université du Botswana, pour l'obtention d'un Certificat d'éducation des adultes. Ce programme, d'une durée de deux ans, se compose de matériel présenté par écrit et de week-ends d'étude mensuels auxquels les étudiants sont tenus d'assister. Le cours s'adresse à des enseignants pour adultes travaillant déjà dans ce domaine, et l'université accepte les candidats possédant le "Junior Certificate" (c'est-dire trois années d'instruction au niveau secondaire). Cette évaluation s'intéressait particulièrement à deux aspects du programme: l'administration de l'enseignement à distance et le contenu des cours. Les informations ont été recueillies par l'intermédiaire de rapports fournis par les assistants au sujet des week-ends d'étude, de discussions de groupes et de questionnaires.
Les résultats de l'enquête ont été codés et analysés au moyen de tabulations combinées, chi au carré et tests R de Pearson, pour déterminer si les différences apparentes étaient signficatives (à des seuils de signification de 0.5* et de 0.0**). Un certain nombre de variables ont influencé les succès académiques, parmi lesquels la garde des enfants,* les heures quotidiennes de travail,** le temps alloué pour l'étude par les employeurs,** la disponibilité et la fréquentation des bibliothèques,* la lecture des journaux,* et l'âge.* Cet article examine aussi la perception des étudiants sur l'utilité du cours, l'efficacité des méthodes d'enseignement employées lors des week-ends d'étude ainsi que le contenu et la qualité du cours. Plusieurs recommandations sont apportées, entre autres la nécessité pour l'Institut de formation des adultes (IAE) de continuer à négocier avec les employeurs pour l'allocation d'un nombre raisonnable d'heures d'étude pendant les heures normales de travail, la révision et l'amélioration des méthodes de soumission des travaux mensuels, l'adaptation des consultations individuelles aux besoins d'apprentissage particuliers des stagiaires et une révision substantielle des unités d'apprentissage trop difficiles.
Introduction
This introduction is intended especially for readers unfamiliar with Botswana as a country; it also serves to outline the main educational structure, objectives, and priorities of the University of Botswana, where non-formal education is seen as an important ingredient in the country's educational system.
Botswana, located in Southern Africa, attained independence from British rule on September 30, 1966. Politically the country is a non-racial democracy which maintains freedom of speech, freedom of the press, and freedom of association. Geographically the country is landlocked; it has a total area of 582,000 km2 and shares borders with Zimbabwe, the Republic of South Africa, Namibia, and Zambia. The climate in Botswana is mainly arid or semi-arid with poor rainfall. Drought is a recurring hazard, and arable land is scarce (only 5% of land is cultivable). Beef and diamonds are the only major natural resources.
Botswana`s population is growing at 3.4% per annum; thus there is a high proportion of children and young people. The total population is estimated at one million people, which is relatively small considering the size of the country. In addition, a high proportion (25% in 1981) of men aged 20 to 40 years are working abroad, mostly in South African mines. Therefore, women predominate among adults and as heads of households (about 34%). The majority of people live in scattered rural settlements and remote areas (only
17.7% of the population are urban dwellers). Economically, the country has four main planning objectives: rapid economic growth, social justice, economic independence, and sustained development. Accordingly, the government has given highest priority to productive employment creation and rural development and training through both the formal and non-formal systems. Botswana, like any third world country, suffers from the constraints of underdevelopment: the lack of and lag in socioeconomic infrastructure and especially in human resource development. Educated workers are scarce and the country still depends on skilled expatriates, although a localization policy is in force.
Currently, the educational system consists of 9 years of primary education, 2 years of junior secondary education, and 2 years of senior secondary education. The country has one national university, and technical and vocational education are still in their infancy. In short, 7 to 9 years of basic education is the greatest extent of formal education available in many schools. Eighty-five percent of pre-school children are in schools, but only about one-third of primary school students gain access to a secondary education. Consequently, the country's educational objectives and priorities include meeting the economic need for trained workers, increasing educational opportunities for all age groups, and providing continuous access to education at all levels.
Since 1978, non-formal education activities have been directed towards correspondence courses at the secondary school level, including the National Literacy Programme and Home Economics. Up to 1984, the correspondence unit enrolled 9,506 students. The National Literacy Programme set a target of 50,000 learners per year from 1980-1985. These non-formal education activities are expanding beyond reading, writing, and numeracy; the needs of rural communities in terms of functional skills required for income- generating activities continue to form the basis for educational opportunities. Thus there is a dire need to train staff for specific job needs, and at present general adult education cannot be emphasized.
But some education is available: apart from the role of the Department of Non-formal Education in training adult educators, the University of Botswana offers courses in adult education at four levels. These are:
- a certificate in adult education (2 years),
- a diploma in adult education (2 years
- a B.Ed. adult education degree (3 years), and
- an M.Ed. adult education degree (2 years).
The candidates for the certificate in adult education are those men and women working in the Department of Non-formal Education and other extension officers, including those from the Prison Department. These people enroll with the university for the certificate to enhance their educational opportunities while on the job. To gauge the effectiveness of the courses, an attempt was made to evaluate the certificate program, and the findings are presented in this article.
The Certificate in Adult Education at the University of Botswana was first offered as a correspondence course in 1983. The students, who have already earned the Botswana Junior Secondary School Certificate, teach adults the "3Rs" in informal settings such as prisons. In the adult education course, students use correspondence texts and mail their completed assignments to the Adult Education Institute at the University of Botswana. This correspondence program is supplemented by compulsory monthly study weekends: two days of classroom teaching in which students review their past work and are introduced to the new unit and assignments. At the end of each year, students spend one month in residence at the university, preparing for final examinations.
The first group of students graduated in 1985. The second group of 37 students (15 males and 22 females) is now undergoing the course. Unfortunately no evaluation was carried out on the first group of students, but it was decided to conduct a formative evaluation of the second. Formative evaluation normally refers to an assessment conducted during the development of a course, but it may also mean the continuous monitoring of the students' progress and an ongoing evaluation of the educational methods used. Formative evaluation provides a good analysis of the effectiveness of any part of the program and it allows reviewers to scrutinize the assumptions and activities that underlie the program.
The case study presented here is a first step in the planned evaluation of the CAE program. This evaluation is to be completed in 1989 with the graduation of the second group of students. It is based largely on a self-administered questionnaire answered by the students in the second group. The questionnaire focused on the students' two study environments: their residence and their workplace. Included were questions about marital status and children; type of housing and the availability of lighting to facilitate home study; hours of work, and the availability of study time during the workday; and the supportive environment, including the availability of libraries and relevant materials to supplement correspondence materials and the value of help from others in the field.
The first goal of this analysis is to refine the administration of distance education to eliminate or reduce any problems encountered, thus aiding the students' progress by removing any obstacles. For example, students suffer from the delay or loss of assignments sent through the post office. As well, students are often inconvenienced by the monthly study weekend and the yearly month-long residence. Of central concern here is the quality of the students' environment and what they perceive as obstructions to their progress.
Another goal of the study was to evaluate the course content in terms of its understandability and its usefulness to the students' actual work in the field. The questionnaire included questions concerning the usefulness of the course to the students' present work and the students' perception of the difficulty of various units. In addition, an attempt was made to correlate the students' answers to their actual performance in terms of their scores in the assignments, tests, and examinations.
Course Evaluation
Overall, most of the students (27 or 73%) reported the program to be of great help to their job, and only one student saw the program as only somewhat useful. The relationship between the students' evaluation of each unit and their actual performance in the course was particularly important. In the first year, students take two main courses. The first is Learning and Teaching in Adult Education, which is divided into six units:
- The Psychology of Adult Education
- Learning Theory and Teaching Methods
- Assessing Learning Needs
- Selecting Appropriate Teaching Strategies
- Learning Materials and Teaching Aids
- Evaluating Teaching and Learning
The second course, Professional English, has five units:
- Structure and Punctuation
- Study Skills
- Summarizing and Note Taking
- Analyzing
- Writing of Reports and Business Letters
In Learning and Teaching in Adult Education, the survey results showed no strict correlation between perception and actual performance (see Table 1). Units 1, 2, and 3, for example, are all essentially theoretical courses but are perceived differently by the students because of the different ways they are presented. Actual performance also varies. For example, Unit 1 is rated "easy," but 21 people scored below 60%, 6 of whom scored 49% or below. In this instance, it appears that the text needs to be supplemented by other reading materials to help students grasp difficult psychological concepts.
In other examples, Unit 2 is rated difficult but performance is high, with 28 students scoring higher than 60%. It may be that on first sight the texts looked difficult but because the students found the course valuable for their actual work they paid more attention to it. Unit 3 is indeed difficult for students, and there is a direct correlation between perception and performance: 22 people ranked in the 50-59% range and 8 in the 49% and below range. There is need in this case to re-examine the course in terms of both content and presentation.
Performance on Units 4 to 6 is relatively high. For example, 28 people scored 70% and above in Unit 5, and 16 scored 70% and above in Unit 4. These scores likely relate to the practical nature of the units. Unit 5, for example, deals with learning materials and teaching aids, touching on issues the students deal with on a daily basis.
Finally, it should be noted that there is a marked difference in performance in theoretical subjects (where performance is relatively low) as opposed to practical subjects (where performance is relatively high). This is understandable since it is not easy to grasp theoretical material while working alone. However, something must be done to facilitate understanding of these subjects. One alternative is to concentrate more on these subjects during the monthly study sessions.
Environmental Effects on Student Performance
A number of assumptions were made regarding the effect of the environment on the students' progress and performance. It was assumed, for example, that the availability of supplementary reading materials would have a significant effect on performance, particularly in theoretical courses. We tried to test this assumption by asking students whether they had had any supplementary reading. Unfortunately, almost 50% of the students did not respond to this question. However, of the 19 students who did respond, the results showed that supplementary readings resulted in better performance. The x2 value was 41.7 with 12 degrees of freedom which resulted in x2 = 0.00%.
It was also assumed that access to libraries and the availability of newspapers would greatly help the students in practical courses. In particular, there appears to have been a significant correlation with regard to the course on Learning Materials and Teaching Aids (x2 = 2.85%). Other assumptions were that the length of the working day affected one's ability to study, and thus one's academic performance, by leaving more or less time for private study. Interestingly, the results here did not show any significant correlation. However, those who got time off for study during working hours did perform better on average than those who had to find their own time to study (x2 =
1.17%). It was equally assumed that the availability of lighting and the type of light used - electricity, kerosene lamps, or candles, and so forth - would affect a student's performance since it would affect reading. The results confirmed this, especially for students using candles, whose average marks ranged between 50% and 59%.
Results may have been negatively affected by the fact that variables were not evenly distributed. For example, while a student may have had time off to study, he or she might not have had access to a library or to supplementary reading materials. Equally, somebody with supplementary reading material might not have had time off to study or had good lighting.
Conclusion
In general, recommendations for improving the administration and content of these and other correspondence courses included the following:
- The University should supply extra reading materials to students, particularly for theoretical subjects. Correspondence texts should include suggested supplementary reading materials, and arrangements should be made either to photocopy some of those materials and provide them with the text or to enable the students to borrow materials from the university library or from the national library network.
- Correspondence courses are taken as a private study and the onus is upon the employer to permit study time during working hours. Consequently, the institute must assist students to secure time off from their employers. The only way that this can be done is for tutors to make visits to the students' workplaces. This has the added advantage of enabling tutors to understand the environment in which the students study.
- The provision of light is important since in enables the student to study at night. Of the 37 students surveyed, only 12 had electricity and another 12 used kerosene lamps. Nine students used candles, which are not good for reading and may damage the eyes. Two others used gas lamps. Students should be encouraged to use adequate lighting. Perhaps employers of students may provide loans so those using candles may buy gas or kerosene lamps.
As noted above, this is a formative evaluation of a relatively new program. At present, only the second group of students is going through the course. Therefore the results are tentative. However, it is important for IAE course administrators to identify and understand problems such as those listed above. For this reason, it is suggested that a questionnaire be administered to all students so that particulars of their environment are known in advance and proper measures may be taken to facilitate learning. The questionnaire should be given to the students when they enrol, and the institute should assist those with problems, especially by negotiating with their employers. Finally, this evaluation should be ongoing, with surveys done annually in both the first and second year of study.
References
Lien, A. J. (1967). Measurement and evaluation of learning. Iowa: William C. Brown.
Neil, M. W. (Ed.). (1981). Education of adults at a distance. Report of the Open University's 10th anniversary international conference. London: Kogan Page.
Taylor, C. (1978). Designs in formative evaluation. In C. T. Fitz-Gibbon & L. L. Morris (Eds.), How to design a program evaluation (pp. 14-18). London: Sage Public.
Weiss, C. H. (1972). Evaluation research, methods for assessing program effectiveness. Prentice Hall.
Williams, J. D. (1973). Course-component evaluation part I: Rationale and outline. Journal of Structural Learning, 4, 25-31.
Dr. Johannes Mutanyatta (United Republic of Tanzania) is a Lecturer in Adult Education at the University of Botswana. Formerly he was head of the Department of Curriculum Programmes and Materials Development at the National Literacy Centre in Mwanza, Tanzania. His research interests and publications focus on may aspects of adult literacy and the curriculum in Adult Education. Dr. Mutanyatta received his doctorate from the University of Toronto, where he specialized in Adult Education.
This work is licensed under a Creative Commons Attribution 3.0 License.
 |