A Scheme for Improving Questionnaire Response Rates

Kevin G. Hayes, Ed.D. Professor of Agricultural Communications and Coordinator of Distance Learning Division of Agricultural Sciences and Natural Resources Oklahoma State University, Stillwater
Adrienne E. Hyle, Ph.D. Associate Professor Education Administration and Higher Education Oklahoma State University, Stillwater

Summary

This paper explains a strategy used in an early 1995 survey of U.S. Higher Education Coordinating Boards and the resulting response rate. In an effort to assess the impact of existing state coordinating board policies on the use of telecommunications for delivering distance education opportunities, a questionnaire was forwarded to the executive officers of state coordinating boards in the 50 states and the District of Columbia as identified by the State Higher Education Executive Officers (SHEEO) organization’s membership listing. Relatively conventional strategies were used to further encourage and/or remind nonrespondents following the initial contact. In addition, emerging communications technologies and organizations were employed in the strategy to encourage participa tion by each SHEEO member. The response rate reached 96.07 percent. This paper reviews techniques proposed by recognized authorities, describes the strategic modifications used in the 1995 study, and reports results. Some data is presented in tabular form in a format similar to that proposed by Fink and Kosecoff (1985, p. 78).

Introduction

The development of television technology, especially the ability to transmit signals via satellite, cable, and fiber optic lines, has greatly enhanced and expanded the capability for delivery of postsecondary education and extension courses via electronic means (Brown and Thornton, 1963; Levenson and Stasheff, 1958; Naisbitt, 1984.) These media include but are not limited to: two-way, talk-back television; satellite delivery of courses to business, industry, and educational institutions; and the use of compressed video (Verduin and Clark, 1991). Recent mergers of telephone and cable companies further open windows of opportunity for adaptation of communication technologies to deliver information and education to all who seek access (Hezel Associates, 1993; 1994). Investments in communications technology on the part of colleges and universities may be the key to institutional survival in the next century (Chaleux, 1985; Hezel, 1990; 1991). Nationally, level or reduced state funding for postsecondary education has generated interest in examining the use of telecommunications technology to deliver quality higher education programs and services at a lower cost (Baird and Monson, 1992). Issues associated with the regulation of both communications and education need to be addressed as soon as they arise if the evolution of higher education is to keep pace with the revolutionary changes in communications technology (Kahin, 1994). In relating public policy to the meaning of educational change, Floyd (1982) believes the ultimate effect of higher education policies directing the use of emerging technologies for distance education should be “maintaining the quality of educational services and providing access to all students who might benefit” (p. 39). Fullan and Stiegelbauer (1991) argue that “strong commitment to a particular change may be a barrier to setting up an effective process of change” (p. 95); that those who wish to see a change implemented need to be as skilled in the change process as they are in the proposed change. These authors demonstrate a need to associate any proposed change with specific goals and objectives and contend that solutions result from shared meaning: “The interface between individual and collective meaning and action in everyday situations is where change stands or falls” (p. 5).

Statement of the Problem

The availability of communications and computer technologies has expanded dramatically throughout the world since the 1950s and currently offers unlimited potential for institutions of higher education to deliver credit and noncredit instruction and information to any part of the world (Naisbitt, 1984). Yet, the delivery of postsecondary education via telecommunications technology is extremely limited in the United States and is more common in cooperative extension work and professional training for adult learners (Baird and Monson, 1992). Another factor limiting the actual use of telecommunications for delivery of postsecondary education is the existence of policies that may discourage colleges and universities from using these technologies. For public institutions of higher education, which are bound by rules and regulations for delivery of postsecondary educational courses, a lag between the availability of technology and its appropriate use is accentuated by state policies that either promote or inhibit deployment (Eure, Goldstein, Gray, and Salomon, 1993). Verduin and Clark (1991) speculate that this anomaly exists because of “American educators’ lack of awareness about just what distant education is, how it operates, and what it can do for adult learning” (p. xi). Fullan and Stiegelbauer (1991) would attribute the time lag between the availability of emerging technologies and their application to needs in higher education to the complexity of the change process. They contend that “educational change is technically simple and socially complex” (p. 65). To better understand this complexity, they posit several assumptions about change; among them are the assumptions that change will take time and “that any significant innovation, if it is to result in change, requires individual implementers to work out their own meaning” (p. 106). In higher education, that meaning is often defined through policy.

Purpose and Scope of the Study

The purpose of the research is to describe the distance education policies and regulations now in place at the state level throughout the United States, and to describe perspectives of the state coordinating board leadership regarding the meaning of these policies to telecommunications-related issues and opportunities facing postsecondary education. Specifically, this descriptive research (Gay, 1992) illuminates perceptions of meaning associated with distance education policy in higher education within the context of Fullan and Stiegelbauer’s Meaning of Educational Change (1991). This study brings focus to the role of coordinating boards in the design, implementation, and coordination of policy which is intended to bring meaning to change. State higher education executive officers throughout the United States and the District of Columbia were surveyed in conducting this assessment of the distance education policies and regulations now in place at the state level throughout the United States. The study was undertaken to gain insight into the perspectives of the educational leadership regarding the relevance of these policies to telecommunications-related issues and opportunities facing postsecondary education. This paper describes the research design through which the study was conducted. The presentation is as follows: (1) definition and selection of population, (2) research instruments, and (3) the procedures used in data collection and analysis.

Definition and Selection of Population

Due to the focused scope of this study, the total population of state higher education executive officers throughout the United States and District of Columbia was included. The population was defined by the then current (January 1995) listing of these officials as provided by the State Higher Education Executive Officers (SHEEO) organization. Forty-nine states, the District of Columbia, and Puerto Rico were members of SHEEO in January 1995. Wyoming is not a member of SHEEO because there is only one four-year institution within that state’s geographic boundary. Because of the effort to obtain data from each state, the Community College Commission in Cheyenne, Wyoming was added to the population. Puerto Rico was not included in the population. In states where more than one individual was listed on the SHEEO mailing list, the first listing was selected solely on the basis of alphabetical prominence.

Research Instruments

Two separate instruments were used in this study: (1) a printed questionnaire distributed to the SHEEO members through FedEx, and (2) a follow-up questionnaire used in telephone interviews with the nonrespondents.

Questionnaire

The questionnaire was developed according to guidelines presented in Bradburn and Sudman (1988), Dillman (1978), and Fowler (1984). Considerable thought was given to the nature of mail questionnaires in regard to “openended” questions. Fowler states without hesitation that “self administered questionnaires should be limited to closed answers” (1984, p. 103). Dillman cautions:

The absence of an interviewer puts the mail questionnaire at a very distinct disadvantage. Not only do some people find it more difficult to express themselves in writing than orally, but the absence of the interviewer’s probes frequently results in answers that cannot be interpreted and sometimes in no answer at all. The difficulty of open-ended questions and the near impossibility of solving it represents one of the most severe shortcomings of mail questionnaires (p. 58).

Bradburn and Sudman (1988) assert that “Mail questionnaires can be effectively used with respondents who have experience in dealing with written materials and above-average motivation to participate in the surveys” (p. 103). These authors consider the principal problem in the design of mail questionnaires to be making questions unambiguous, so respondents do not need to puzzle over a question’s meaning. Assumptions which need to be noted here are that the population contacted for this study consists of individuals: (1) who are experienced with written materials, (2) can express themselves in the written word as well as orally, and (3) have high motivation to participate in a survey dealing with the topic at hand.

Development

Ultimately, the questionnaire was designed with both open and closed questions. Questions focused on sources of innovation, the meaning of change, and factors affecting implementation and continuation of delivering postsecondary education via telecommunications. A draft copy of the questionnaire was then sent to nine reviewers in six states who are knowledgeable about and interested in distance education at the postsecondary level. The panel consisted of two academic deans, two directors of continuing or extension education, two professors currently serving as coordinators of distance education at their respective land-grant universities, the academic vice president of a major distance education consortium, a professor of statistics, and a professor of education. All but one of the panel members responded. Each had recommendations for improving the instrument; none suggested eliminating the open-ended questions. Once the questionnaire was rewritten to reflect the panel’s recommendations, the data collection procedure began.

Data Collection Procedure

The strategy employed for encouraging response is a minor modification of Fowler’s (1984) scheme for reducing nonresponse to mail surveys through repeated contact (p. 54) and is presented in Table 1. Nine days after the initial mailing, a post card was sent to nonrespondents. The message on the card reminded the recipient that a questionnaire had been sent on January 16, that a FedEx PAK had been provided for expense-free return of the questionnaire and related documents, and that we placed high value on a response from each state. The investigator’s return address, phone number, and FAX number were included to make it easy for recipients to make contact in the event they had not received the material, had misplaced it, or had questions about the survey. One week later (February 1, 1995), a personalized letter of endorsement for the study was sent to each nonrespondent from Dr. Hans Brisch, Chancellor of the Oklahoma State Regents for Higher Education, on official stationery from his office. Dr. Brisch’s letter spoke of the potential value of the data and noted that such value increases with the response rate.

Ten days later, the date participants were initially asked to return the questionnaire and related documents, another mailing was prepared for nonrespondents which included a letter advising the recipient of the importance of each state’s participation, but also restating that participation was voluntary. This mailing, which was sent via the U. S. Postal Service in a green-and-white envelope, included another copy of the questionnaire--with the deadline date changed (in red ink) to February 24, 1994--and a return envelope with two First Class postage stamps affixed. The response rate reached 78.4 percent prior to implementation of the telephone follow-up on March 6, 1995 , increased to 92.15 percent by March 14, 1995 (the announced final deadline), and to 96.07 percent by mid-summer (Table 1).

Table 1. A Scheme for Reducing Nonresponse

This response rate should be considered better than average, based on Fowler’s (1984) observations:

The difficulties of getting the response rate to a reasonable level will depend on the nature of the sample, the nature of the study, how motivated people are, and how easy the task is for them. Clearly, the task will be easier if the sample is composed of motivated, well-educated individuals. However, Dillman has obtained response rates over 70 percent with general population samples, using only mail procedures (Dillman et al., 1974). Hochstim (1967) obtained response rates over 80 percent with telephone and personal follow ups. If the researcher will be persistent, and if it is a reasonably well conceived and well-designed study, acceptable response rates can be obtained by mail. (p. 55)

Several attempts were made to attract the recipient’s attention and to imply urgency through the initial mailing. All communications were personalized for each recipient. The initial letter and questionnaire were sent by FedEx to arrive on January 17, 1995. The cover letter was prepared on stationery from the Office of Academic Programs, College of Agricultural Sciences and Natural Resources, Oklahoma State University. The questionnaire was printed on colored paper to help keep it more visible on the recipient’s desk and a FedEx PAK was included with the first mailing to help facilitate the respondents’ return of all documents at no expense to their respective offices.

Bradburn and Sudman (1988) recommend a well-written cover letter, multiple mailings, and “a small monetary gift, usually a dollar” (p. 104) to increase motivation to respond. The only “gift” offered participants in exchange for completing and returning the questionnaire was a copy of the summary information at the conclusion of the study (Appendix B).

The post card reminder, mailed on January 25, 1995, provided another opportunity to attract attention because postage rates had just increased, necessitating the addition of a colorful one-cent stamp to the already colorful U.S. Postal Service card. Both the message and mailing surface were imprinted directly on each card using a desk-top laser printer.

After reviewing information on constructing telephone questionnaires (Dillman, 1978), the instrument used for the mail survey was adapted for telephone use. Eleven nonrespondents were identified following the final mail delivery on March 3, 1995 (one week after the “adjusted” deadline of February 24, 1994), and the telephone contact began the following Monday, March 6, 1995.

Respondents

Responses from 48 states plus the District of Columbia were received. The states represented by the respondents are geographically diverse, including both continental and non-contiguous representation (Table 2). States not responding are Iowa and Maine. The 49 respondents provided 46 completed questionnaires and one completed telephone interview; two of the respondents returned the questionnaires without completing them.

As of March 6, 1995, 40 responses had been received (Table 1, page 41). Offices of the 11 nonrespondents were then contacted by telephone. Three individuals indicated work on the questionnaire was still in progress and that the questionnaire would be returned; six offices promised a return call; one individual indicated she had put the questionnaire in the mail on March 3, 1995; and one consented to a telephone interview after stating he did not intend to complete the questionnaire. Ultimately, the telephone follow-up effort resulted in an overall response rate of 96.07 percent. The coordinating boards participating in the study and their modes of response are identified in Table 2.

Table 2. Coordinating Boards Participating in Study

References

Baird, M.A., and Monson, M.K. (1992). Distance education: Meeting diverse learners’ needs in a changing world. New Directions for Teaching and Learning, 51:65-76.
Bradburn, N.M., and Sudman, S. (1988). Polls and surveys: Understanding what they tell us. San Francisco: Jossey-Bass.
Brown, J.W., and Thornton, J.W., Jr. (1963). New media in higher education. Washington, DC: National Education Association.
Chaloux, B.N. (1985). The project on assessing long distance learning via telecommunications: Project alltel. Denver: Council on Postsecondary Accreditation, State Higher Education Executive Officers. Dillman, D.A. (1978). Mail and telephone surveys: The total design method. New York: John Wiley & Sons.
Dillman, D.A., Carpenter, E., Christensen, J., & Brooks, R. (1974). Increasing mail questionnaire response: a four state comparison. American Sociological Review. 39(5):744-756.
Eure, C.A., Goldstein, M.B., Gray, T.D., and Solomon, K.D. (1993, September). The distance learning regulatory and policy environment. Proceedings of a seminar held on behalf of the Oklahoma State Regents for Higher Education and Oklahoma State University, Stillwater, OK.
Fink, A., and Kosecoff, J. (1985). How to conduct surveys: A step-by-step guide. Newbury Park, CA: Sage.
Floyd, C.E. (1982). State planning, budgeting, and accountability: Approaches for higher education. AAHE-ERIC/ Higher Education Research Report No. 6. Washington, DC: American Association for Higher Education. (ERIC Document Reproduction Service No. ED 182 465)
Fullan, M.G., and Stiegelbauer, S. (1991). The new meaning of educational change. (2nd ed). New York: Teachers College Press.
Fowler, F.J., Jr. (1984). Survey research methods. Beverly Hills: Sage.
Gay, L.R. (1992). Educational research: Competencies for analysis and application (4th ed.). New York: Macmillan Publishing Company.
Hezel Associates. (1993). The time is now! Teleconferencing in state government: 1993. Washington, DC: The International Telecommunications Association.
Hezel Associates. (1994). Educational telecommunications: The state-by-state analysis. Syracuse: Author.
Hezel, R.T. (1990). Policies for educational technology: A national, state, and local agenda. (Paper commissioned for the Chief School Officers’ State Technology Conference in Minneapolis, April 29-May 2, 1990.) Elmhurst, IL: North Central Regional Educational Laboratory. (ERIC Document Reproduction Service No. ED 182 465)
Hezel, R.T. (1991). Statewide planning for telecommunications in education: Some trends and issues. TechTrends, 36(5):17-20.
Hochstim, J.R. (1967). A critical comparison of three strategies of collecting data from households. Journal of the American Statistical Association, 62(September):976-989.
Kahin, B., ed. (1994). Information infrastructure sourcebook (Version 3.0, Vols. 1 and 2). Cambridge, MA: Harvard University.
Levenson, W.B., and Stasheff, E. (1958). Teaching through radio and television. New York: Rinehart & Company.
Verduin, J.R., Jr., and Clark, T.A. (1991). Distance education: The foundations of effective practice. San Francisco: Jossey-Bass.