Vaccination coverage (VC) estimates are essential to monitor the performance of immunisation programmes and take action to improve them. In resource-poor settings, administrative estimates of VC, reached by dividing the number of people vaccinated by the population in the target age group, are often biased due to inaccurate population figures and pressure on programmes to report favourable indicators. Sample surveys are thus frequently employed to establish more accurate estimates.
A specific challenge in these settings is estimation of VC at the local level (e.g. district, sub-district or health catchment area), so as to identify communities that may require additional support (e.g. supplementary campaigns, strengthening of routine vaccination) and allocate limited resources efficiently. To do this, two survey methods recommended by the World Health Organization are available: cluster surveys and lot quality assurance sampling (LQAS) .
Cluster surveys feature simple designs that do not require accurate population figures or household sampling frames . However, cluster samples cannot be used to make inferences for individual communities within the sampling universe; therefore, for each community of interest, one independent cluster sample needs to be selected. Typical sample sizes for such cluster surveys are on the order of 30 clusters x 7 individuals . Theoretically even smaller samples may be chosen, but there is insufficient evidence on whether the resulting estimates are likely to be robust, i.e. whether both the point estimate and the estimated standard error (SE) remain acceptably stable as sample size decreases .
LQAS has been promoted as a faster and cheaper alternative to cluster surveys for monitoring various public health interventions , though it could be potentially misused due to erroneous statistical assumptions . In this approach, a random sample N of individuals (or other basic sampling units, depending on the indicator being monitored) is selected within each community, or lot. LQAS yields a binary classification decision: in the case of vaccination, the lot is “rejected” (i.e. judged to require supplementary activities) if the number of unvaccinated individuals within the sample exceeds a decision threshold d, and “accepted” otherwise. Various sampling plans consisting of a given N and d can be used. However, in practice where both time and resources are often limited, one needs to specify a lower threshold VC (LT), i.e. the minimum acceptable VC below which supplementary interventions (e.g. re-vaccination) must take place; and an upper threshold VC (UT), usually fixed at the target VC. Each sampling plan features a probability α that the survey will yield an acceptance decision when in fact the lot has a VC < LT (this is known as the “consumer” risk, as it deprives beneficiaries of the intervention they need); and a probability β that the lot will be rejected when in fact VC exceeds the UT (this constitutes the “provider” risk of expending resources needlessly). Minimising the consumer risk is the main criterion for selecting a sampling plan. Minimising the provider risk is also important, but in many situations a relatively high provider risk is tolerated so as to ensure that the resulting sample size still makes LQAS more advantageous than a standard survey. The combination of a large grey zone between LT and UT (a result of the sampling plan) and a high proportion of communities falling within this grey zone (a phenomenon independent of the sampling plan, but merely reflecting how the variable is distributed in the population) however results in a high classification error .
The theoretical advantage of LQAS is that it yields the desired information with much smaller sample sizes than cluster surveys. However, the often-overlooked requirement for a fully random sample poses a serious challenge in resource-poor settings, since updated lists of households are often unavailable, and since random sampling will usually require travel to an unfeasibly large number of sites within the community.
To overcome this problem, Pezzoli et al. and Greenland et al. [8–10] have recently put forward a more field-friendly “clustered LQAS” (CLQAS) approach, whereby the lot sample is divided into clusters, as in any multi-stage cluster sample. The critical assumption behind this approach is that, within any given lot (e.g. a district), the true VC levels in the different individual primary sampling units (e.g. villages), among which one would randomly select clusters, always give rise to a binomial distribution, with the mean of this distribution equal to the overall VC of the lot, and the standard deviation equal to or less than an a priori assumed level. The authors propose various sampling plans (e.g. 5 clusters of 10 individuals) that, for assumed standard deviations ≤ 0.05 or ≤ 0.10 and typical LT and UT thresholds of interest, yield reasonably low α and β probabilities.
As estimates of local vaccination coverage are used to orient subsequent catch-up vaccination activities, the choice of appropriate survey methodology is essential. The CLQAS approach has been used in different settings including Nigeria and Cameroon [8, 9]; however, the accuracy of classifications generated by this design and implications of this accuracy for operational decisions have not been sufficiently documented . Using data from a vaccination coverage survey carried out in Mali in January 2011, we aimed to evaluate the performance of CLQAS in a typical field setting. We also explored whether classical surveys using smaller samples than currently recommended provide results that, although less precise, are still statistically stable and useful for operational decision-making, and could thus constitute an alternative to CLQAS.