An attempt has been made to apply a model for evaluating program effectiveness and efficiency to an actual program on a retrospective basis. It was not possible to apply the model fully since there was a lack of specification and measurement of program sub-objectives.
In the absence of a predetermined “extent of intended attainment,” program effectiveness was computed using a Utopian, perhaps unrealistic standard of total elimination of violations. Two different measures of program objectives were constructed from available data. The first is a rating scale similar to the conventional rating scales recommended by the Public Health Service for surveying food and milk control programs. It differs primarily in that a given category may have variable debits assigned, based on judged severity of the conditions as compared to the minimum acceptable standard. The second measure used was correction rate, that is, the proportion of violations recorded at a given inspection that were corrected (not in violation) at the time of the subsequent inspection.
Both measures indicate the program was operating in a maintenance fashion during the period studied. The difference in the average ratings of establishments studied were not significantly different before and after the period. Although a considerable number of violations were corrected during the period, a slightly greater number of new violations occurred. Observations of a district that was not inspected for a period of time suggests that ratings would decline in the absence of the program.
Program activities were performed inconsiderably less quantity than planned but at near planned quality. Correlation, of activity with objectives indicates that the number of activities performed has a small positive correlation with attainment of objective but that quality of activity as measured has a negative correlation with attainment. It was impossible in the present study to determine whether the assumptions about the quality of work that is thought to be effective are in error or whether the measure of quality was invalid. The analysis leads one to believe that had all activity been performed as planned, effectiveness would have been greater but still considerably short of complete attainment.
Program resources were expended at very nearly the planned level. Thus, failure to perform the number of activities planned resulted from error in planning assumptions linking these variables, not from failure to use resources as planned.
Efficiency was considerably less than anticipated for all efficiency measures.
It would seem four general directions are available to the program director in terms of future operations.
1. Since there is considerable room for improvement, both in terms of rating and correction rates, he may plan to increase program effectiveness in the future. The implications from this study are that some improvement would be possible by performing more activity of the same sort as in the past. It would seem advisable, however, to think more in terms of modifying the qualitative aspects of activity. Unless some basis could be found for justifying a particular modification, it might be preferable to experiment with several kinds of qualitative modifications in different areas to learn the relative effectiveness (and efficiency) of different kinds of activity. Current efficiency findings would be of little value in planning for the future should different kinds of activity be planned.
2. The program director might decide that current level of effectiveness and efficiency are about as high as possible, and are satisfactory. In this instance, he would be likely to decide to maintain current levels of operation.
3. A third kind of decision would be to maintain similar levels of objectives while trying to improve current efficiency, levels.
4. Finally, based on knowledge of current levels of operations and estimates of potential alternatives, the decision could be made to discontinue the program in favor of some equally beneficial alternative program where efficiency is estimated to be greater. Since the program is established on a legislative rather than administrative basis, such a decision may be unrealistic in the present case.
It does seem clear, however, that considerations of desired and actual effectiveness should logically precede considerations of efficiency. The planner would consider first the levels of effectiveness that are desirable and only then consider the expected costs of each level of achievement. His final decision concerning future operations will depend on the relationship of costs to predicted benefits, except in the rare case where an objective is deemed so desirable that costs are of no concern.
1This investigation was supported by the Public Health Service, Research Grant No. CH 00044 from the Division of Community Health Services.