What’s more general than a whole population?
© Alexander. 2015
Received: 30 June 2014
Accepted: 22 May 2015
Published: 25 August 2015
Statistical inference is commonly said to be inapplicable to complete population studies, such as censuses, due to the absence of sampling variability. Nevertheless, in recent years, studies of whole populations, e.g., all cases of a certain cancer in a given country, have become more common, and often report p values and confidence intervals regardless of such concerns. With reference to the social science literature, the current paper explores the circumstances under which statistical inference can be meaningful for such studies. It concludes that its use implicitly requires a target population which is wider than the whole population studied — for example future cases, or a supranational geographic region — and that the validity of such statistical analysis depends on the generalizability of the whole to the target population.
If Czech history could be repeated, we should of course find it desirable to test the other possibility each time and compare the results. Without such an experiment, all considerations of this kind remain a game of hypotheses .
One way to preserve sampling-based statistical inference is to adduce a group that is even more general than the whole population that was studied. This kind of wider group is called here a target population, although the term is sometimes used as a synonym for the sampling population . The target population is the group ‘about which conclusions are to be made’  but, when it differs from the sampling population, it is not subject to the sampling variation formalism. Some authors define target populations to be real  (p361) while for others they do not necessarily exist, at least not yet. For example, Kirkwood and Sterne say that enumeration of a target population may be impossible because it ‘often includes not only all persons living at present but also those that may be alive at some time in the future’  (p10). Despite being currently intangible, such target populations can be important to the application in hand, the cited authors’ example being future recipients of an experimental vaccine. Target populations (in this sense) are sometimes called superpopulations, which comprise ‘all possible persons that ever were or ever could be targets of inference’ .
In the following sections we look at attempts to retain statistical inference in whole population studies, then argue that the problem is better seen as one of generalizability.
The bootstrap generates variation by sampling the original data with replacement. This can actually be carried out in practice, unlike the notional repeated sampling of a population. However the validity of the bootstrap depends on ‘independent identical sampling from an unknown distribution’  so the original data are themselves still assumed to result from a sampling process.
Consider the following two examples of a binary outcome with complete population coverage
Presidential election: there is 100% turnout in a national presidential election, with each vote being either for candidate A or candidate B.
Cancer registry: male or female sex is registered for all cases in a national registry which has 100% coverage of the type of cancer in question.
In the election example, is it meaningful to estimate a sampling error for the proportion voting for candidate A? The answer seems to be clearly ‘no’. This is because the purpose of the election is to choose a president, which is done on the basis of the observed proportion of votes cast. Any kind of interval estimate serves no purpose because there is no generalizability beyond the election.
In the cancer registry example, is it meaningful to estimate a sampling error for the proportion of cases who are female? Some would say ‘no’ on the basis that it’s a complete population enumeration with no sampling error. Similar examples in the literature show that some authors would say ‘yes’. This implies an attempt to generalize beyond the population observed, but what is this wider target population? Conceivably future cases, or a wider, supranational geographical area, although often this is left unspecified.
Representativeness and generalizability
The above reasoning suggests that the validity of either frequentist or Bayesian inference depends on the extent to which the group being studied — whether or not it is a whole population — is representative of the target population. Porta  allows a sample to be representative of a population if it is ‘typical in respect of certain characteristics, however chosen’, i.e., without requiring a particular selection method such as random sampling. Hence a study group can potentially be representative of a given target population, and its findings generalizable to it, even if the target population is not amenable to sampling.
The current paper argues that the problem of inferential methods in whole populations is most usefully understood as one of generalizability (Fig. 2). Thinking in these terms helps show that inferential methods are meaningless for some situations while for others their use is at least arguable (see Table 1).
Lack of information on generalizability has been identified as a limiting factor in the translation of research findings to policy [17–19]. The CONSORT guidelines for clinical trials , and STROBE for observational studies  mandate discussion of generalizability but give no guidance on how to assess it. Point 21 of STROBE, for example, is ‘Discuss the generalisability (external validity) of the study results’. Several frameworks for assessing generalizability have been proposed [17, 22–24]. Inferential analysis of whole population studies is justified only as far as such studies are generalizable, yet reporting of generalizability in other kinds of study is often poor [18, 19]. The extent to which this also applies to whole population studies is briefly assessed in the following section.
Literature review on generalizability of whole population studies
A literature review was carried out to assess the extent to which whole population studies assess their own generalizability. The search term ‘whole population [TI]’ was used in PubMed on 25 February 2015, restricted to publications in 1994 or later, yielding 64 publications. They were retained if containing primary data on studies of humans, and reporting p values or confidence intervals in the abstract. The full text of each of the resulting 13 studies was reviewed for description of its generalizability or external validity (Table 2). The populations were typically either whole countries (e.g., Iceland) or subnational administrative regions of different sizes (e.g., Western Australia, Isle of Wight). Only two papers [25, 26] made reference to wider target populations to which the results might be generalized, and which might give meaning to the p values and/or confidence intervals used. These findings can only be suggestive because a limitation of the search is that it does not cover all studies which the current paper would call ‘whole population’. For example, the previously-cited Danish National Acute Leukemia Registry  has estimated coverage of more than 98 %, and a PubMed search for its name yields 11 studies, but none of these met the current search criteria. Among the studies which were identified, however, few are concerned about their generalizability despite the fact that, on the reasoning of the current paper, the validity of their statistical inference depends on it. In particular none of them mentioned the STROBE guidelines which mandate that generalisability be considered.
Assessment of generalizability or external validity
‘We are confident about the generalisability of our analytic findings on associations with stimulant medication use Australia-wide’
North East Scotland
‘Our results could be generalisable to young children across the UK'
‘Top End’ of the Northern Territory of Australia
Isle of Wight, United Kingdom
Isle of Wight, United Kingdom
Whole population studies often calculate p values and confidence intervals which have no explicit theoretical basis. This is a problem that social sciences have, perhaps, confronted more directly than epidemiology [9, 27]. Having concluded that purely statistical workarounds are futile, should we eschew inferential statistics altogether and concentrate instead on descriptive statistics ? The current paper advocates a less absolute position; that the conclusions of whole population studies are valid to the extent that their study populations are representative of a wider target population. Hence the problem is converted into one of generalizability. In turn this should be manifest in each study’s research question. For example, who is intended to benefit from analysis of national cancer registry data to be applied; future cases in the same country, and/or those further afield? If the former, making this explicit may highlight needs for particular statistical methods, such as time series analysis on the existing data disaggregated by time. Some target populations may not be possible to sample, e.g., because they lie in the future. Similarly, causal inference deals with unobserved or counterfactual outcomes [29, 30], and can be cast in terms of target and source populations , and is a promising approach for analysis of whole population studies.
In epidemiology, although aspirations to generalizability are not always met, groundwork has been done on its objective assessment [17, 22–24, 32]. For inferential statistics of whole population studies to be more meaningful, they should fully comply with existing guidelines on reporting generalizability [20, 21], and these guidelines should themselves be updated to reflect the developing best practice.
I receive support from the United Kingdom Medical Research Council (MRC) and Department for International Development (DFID) (MR/K012126/1). I am grateful to Karim Anaya-Izquierdo for introducing me to the field of generalizability, and to him, Lyda Osorio, and anonymous referees for helpful discussion of an earlier version of this paper.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Kundera M. The Unbearable Lightness of Being. London: Faber and Faber; 1984.Google Scholar
- Armitage P, Berry G, Matthews JNS. Statistical Methods in Medical Research. 4th ed. Oxford: Blackwell Scientific Publications; 2001.Google Scholar
- Snedecor GW, Cochran WG. Statistical Methods. 8th ed. Ames: Iowa State University Press; 1989.Google Scholar
- Fathalla MF, Fathalla MMF. A Practical Guide for Health Researchers. World Health Organization Regional Office for the Eastern Mediterranean: Cairo; 2004.Google Scholar
- Burr JM, Mitry E, Rachet B, Coleman MP. Survival from uveal melanoma in England and Wales 1986 to 2001. Ophthalmic Epidemiol. 2007;14(1):3–8.PubMedView ArticleGoogle Scholar
- Hakonarson H, Gulcher JR, Stefansson K. deCODE genetics. Inc Pharmacogenomics. 2003;4(2):209–15.PubMedView ArticleGoogle Scholar
- Ostgard LS, Norgaard JM, Severinsen MT, Sengelov H, Friis L, Jensen MK, et al. Data quality in the Danish National Acute Leukemia Registry: a hematological data resource. Clinical epidemiology. 2013;5:335–44.PubMedView ArticlePubMed CentralGoogle Scholar
- Barlow L, Westergren K, Holmberg L, Talback M. The completeness of the Swedish Cancer Register: a sample survey for year 1998. Acta Oncol. 2009;48(1):27–33.PubMedView ArticleGoogle Scholar
- Berk RA, Western B, Weiss RE. Statistical inference for apparent populations. In: Marsden PV, editor. Sociological Methodology 1995. Cambridge: MA: Blackwell Publishers; 1995. p. 421–58.Google Scholar
- Porta M. Dictionary of Epidemiology. 5th ed. Oxford: Oxford University Press; 2008.Google Scholar
- Coggon D, Rose G, Barker DJP. Epidemiology for the Uninitiated. 4th ed. London: BMJ Publishing; 1997.Google Scholar
- Rothman KJ, Greenland S. Modern Epidemiology. 2nd ed. Philadelphia: Lippincott-Raven Publishers; 1998.Google Scholar
- Kirkwood BR, Sterne JAC. Essentials of Medical Statistics. 2nd ed. Oxford: Blackwell Scientific Publications; 2003.Google Scholar
- Bollen KA. Apparent and nonapparent significance tests. In: Marsden PV, editor. Sociological Methodology 1995. Cambridge: MA: Blackwell Publishers; 1995. p. 459–68.Google Scholar
- Efron B. Nonparametric estimates of standard error: the jackknife, the bootstrap and other methods. Biometrika. 1981;68(3):589–99.View ArticleGoogle Scholar
- Bayarri MJ, Berger JO. The interplay of Bayesian and frequentist analysis. Stat Sci. 2004;19(1):58–80.View ArticleGoogle Scholar
- General Accounting Office. Cross design synthesis: a new strategy for medical effectiveness research. Washington, DC: GAO; 1992.Google Scholar
- Dzewaltowski DA, Estabrooks PA, Klesges LM, Bull S, Glasgow RE. Behavior change intervention research in community settings: how generalizable are the results? Health Promot Int. 2004;19(2):235–45.PubMedView ArticleGoogle Scholar
- Klesges LM, Williams NA, Davis KS, Buscemi J, Kitzmann KM. External validity reporting in behavioral treatment of childhood obesity: a systematic review. Am J Prev Med. 2012;42(2):185–92.PubMedView ArticleGoogle Scholar
- Schulz KF, Altman DG, Moher D. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMC Med. 2010;8:18.PubMedView ArticlePubMed CentralGoogle Scholar
- von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ. 2007;335(7624):806–8.View ArticleGoogle Scholar
- Bonell C, Oakley A, Hargreaves J, Strange V, Rees R. Assessment of generalisability in trials of health interventions: suggested framework and systematic review. BMJ. 2006;333(7563):346–9.PubMedView ArticlePubMed CentralGoogle Scholar
- Dekkers OM, von Elm E, Algra A, Romijn JA, Vandenbroucke JP. How to assess the external validity of therapeutic trials: a conceptual approach. Int J Epidemiol. 2010;39(1):89–94.PubMedView ArticleGoogle Scholar
- Watts P, Phillips G, Petticrew M, Harden A, Renton A. The influence of environmental factors on the generalisability of public health research evidence: physical activity as a worked example. The international journal of behavioral nutrition and physical activity. 2011;8:128.PubMedView ArticlePubMed CentralGoogle Scholar
- Smith S, Craig LC, Raja EA, McNeill G, Turner SW. Prevalence and year-on-year trends in childhood thinness in a whole population study. Arch Dis Child. 2014;99(1):58–61.PubMedView ArticleGoogle Scholar
- Ghosh M, Holman CD, Preen DB. Exploring parental country of birth differences in the use of psychostimulant medications for ADHD: a whole-population linked data study. Aust N Z J Public Health. 2015;39(1):88–92.PubMedView ArticleGoogle Scholar
- Western B, Jackman S. Bayesian inference for comparative research. Am Polit Sci Rev. 1994;88:412–23.View ArticleGoogle Scholar
- Ballinger C. Why inferential statistics are inappropriate for development studies and how the same data can be better used. In: MPRA. University Library of Munich, Germany: Paper 29780; 2011.Google Scholar
- Hubbard AE, Laan MJ. Population intervention models in causal inference. Biometrika. 2008;95(1):35–47.PubMedView ArticlePubMed CentralGoogle Scholar
- Pearl J. An Introduction to Causal Inference. Int J Biochem. 2010;6(2):7.Google Scholar
- Höfler M. Causal inference based on counterfactuals. BMC Med Res Methodol. 2005;5:28.PubMedView ArticlePubMed CentralGoogle Scholar
- Post PN, de Beer H, Guyatt GH. How to generalize efficacy results of randomized trials: recommendations based on a systematic review of possible approaches. J Eval Clin Pract. 2013;19(4):638–43.PubMedView ArticleGoogle Scholar
- Malacova E, Kemp A, Hart R, Jama-Alol K, Preen DB. Long-term risk of ectopic pregnancy varies by method of tubal sterilization: a whole-population study. Fertil Steril. 2014;101(3):728–34.PubMedView ArticleGoogle Scholar
- Meuleners LB, Fraser ML, Ng J, Morlet N. The impact of first- and second-eye cataract surgery on injurious falls that require hospitalisation: a whole-population study. Age Ageing. 2014;43(3):341–6.PubMedView ArticleGoogle Scholar
- Kemp A, Preen DB, Morlet N, Clark A, McAllister IL, Briffa T, et al. Myocardial infarction after intravitreal vascular endothelial growth factor inhibitors: a whole population study. Retina. 2013;33(5):920–7.PubMedView ArticleGoogle Scholar
- Clark A, Morlet N, Ng JQ, Preen DB, Semmens JB. Risk for retinal detachment after phacoemulsification: a whole-population study of cataract surgery outcomes. Arch Ophthalmol. 2012;130(7):882–8.PubMedView ArticleGoogle Scholar
- Meuleners LB, Hendrie D, Lee AH, Ng JQ, Morlet N. The effectiveness of cataract surgery in reducing motor vehicle crashes: a whole population study using linked data. Ophthalmic Epidemiol. 2012;19(1):23–8.PubMedView ArticleGoogle Scholar
- Palsdottir HB, Hardarson S, Petursdottir V, Jonsson A, Jonsson E, Sigurdsson MI, et al. Incidental detection of renal cell carcinoma is an independent prognostic marker: results of a long-term, whole population study. J Urol. 2012;187(1):48–53.PubMedView ArticleGoogle Scholar
- Clark A, Morlet N, Ng JQ, Preen DB, Semmens JB. Whole population trends in complications of cataract surgery over 22 years in Western Australia. Ophthalmology. 2011;118(6):1055–61.PubMedView ArticleGoogle Scholar
- Langhorne P, Lewsey JD, Jhund PS, Gillies M, Chalmers JW, Redpath A, et al. Estimating the impact of stroke unit care in a whole population: an epidemiological study using routine data. J Neurol Neurosurg Psychiatry. 2010;81(12):1301–5.PubMedView ArticleGoogle Scholar
- Currie BJ, Jacups SP, Cheng AC, Fisher DA, Anstey NM, Huffam SE, et al. Melioidosis epidemiology and risk factors from a prospective whole-population study in northern Australia. Trop Med Int Health. 2004;9(11):1167–74.PubMedView ArticleGoogle Scholar
- Arshad SH, Tariq SM, Matthews S, Hakim E. Sensitization to common allergens and its association with allergic disorders at age 4 years: a whole population birth cohort study. Pediatrics. 2001;108(2), E33.PubMedView ArticleGoogle Scholar
- Tariq SM, Matthews SM, Hakim EA, Stevens M, Arshad SH, Hide DW. The prevalence of and risk factors for atopy in early childhood: a whole population birth cohort study. J Allergy Clin Immunol. 1998;101(5):587–93.PubMedView ArticleGoogle Scholar