In reality many models utilized multiple parameterization strategies. For instance, combining estimates from the literature, censuses and maximum likelihood. For simplicity, models were categorized by the less common and most sophisticated technique used. For instance, a model using literature estimates and Bayesian inference was categorized as using Bayesian methods for parameterization. Among all models, the dominant parameterization strategy (used by 47% of the models) was to adopt parameters from previous studies, especially from other modelling studies perpetuating the use of parameters chosen by other modellers (Figure 1A “parameterization”). 25% of the studies utilized information or parameters derived from epidemiological, laboratory (e.g. viral shedding duration, cohort studies) or case data (e.g. epidemic curves, attack rates) from other sources to parameterize the model. It was common (60%, Figure 1C) to use some sort of sensitivity analysis and this was more frequent in models that did not directly adopt parameters from previous models, suggesting that sensitivity analysis was not used as a complement to reusing parameters from previous models. ABMs were more frequently built using parameters chosen by modellers in previous studies (70% Figure 1C) and constructed from population demographic data, for instance from decennial censuses, rather than using empirical data or parameters obtained from epidemiological or laboratory studies.

Although the most common approach was to parameterise models using parameter values chosen by previous modelling studies, there were several exceptions that used alternative parameterization methods (Figure 1B shows the distribution of parameterization methods and Table 2 defines the methods) ranging from calibration through simulation [33, 41, 42], maximum likelihood [12, 36], least squares [1, 11] and Bayesian computational methods such as Markov chain Monte Carlo (MCMC) [4] (Figure 1A).

Several real-time pandemic modelling articles involved sophisticated methods of parameterization employing on-going observed case data, such as maximum likelihood estimation [9] or sequential particle filtering within a Bayesian framework [43]. Their real-time nature enabled the possibility of continuous open validation regarding the prediction of pandemic characteristics such as the timing and height of the peak, and indeed Ong et al. [43] report posting real-time predictions on the internet.

There were several non-real-time examples of modelling papers that parameterized compartmental models using disaggregated epidemic data such as: questionnaire or survey results [44, 45]; serological data [36, 46]; epidemic cases or mortality time series [1, 47–49]; and observed time of pandemic peaks [11, 12]. Examples of parameterization from historical epidemic data in ABMs included calibration to reproduce attack or serological infection rates from previous pandemics [33, 41, 42, 50]. Parameterization from case data can be used to investigate policy effectiveness. For instance, Cauchemez et al. [4] evaluated the effectiveness of school closures for pandemic control in France and showed that prolonged school closures would potentially reduce the attack rate of a pandemic by 13–17% by using MCMC Bayesian computational methods to fit an age-structured household-based compartmental model to influenza surveillance data.

Most of the reviewed models reproduced parameter choices from previous studies. This is to be expected as deriving parameters from outbreak data is complex. As a result, some articles specialize in the statistical analysis that leads to parameter derivation and others specialize in the analysis of broad policy questions. There is however the risk that this approach may perpetuate faulty parameterisations from previous studies, or applies a valid parameter value to an inappropriate setting. On the other hand, informing too many parameters in the model by fitting to epidemic time series may run the risk of overfitting or non-identifiability. It may be most credible to inform model parameters using a combination of field or laboratory studies data (e.g. to fit or even directly inform parameters such as recovery rates) and epidemic case data (e.g. to fit transmission related parameters), and then compare fitted parameter values to those obtained from previous studies. One of the possible explanations why this combination of data sources is not common is data paucity, rendering the use of parameters chosen from other modelling studies as one of the few alternatives. One way to increase the pool of available data for model parameterization is to establish international data sharing mechanisms among governments and researchers, especially regarding disease transmission between individuals and surveys of population contact patterns [51], to facilitate the construction of robust models.

Even if epidemic data are available, the small number of models parameterized from such data might also reflect statistical difficulties brought about by censorship in the data—some processes cannot be observed, and many influenza infections are not virologically confirmed, have indistinguishable symptoms, or are asymptomatic. Such censoring combines with non-independence between observations to prevent the use of standard statistical techniques. While such difficulties can be overcome, for instance using maximum likelihood estimation methods [9], particle filtering [43] or other likelihood-based computational methods [4] (Table 3), these require at least some mastery of modern statistical techniques and may be computationally intensive. For instance, Bayesian methods that use MCMC algorithms or approximate Bayesian computation methods can be particularly powerful and flexible tools (Table 2) [52]. These methods allow the merging of prior knowledge on the epidemic parameters—such as those derived from datasets described in the literature—with observed data from the outbreak in question. In addition they allow rigorous parameterisation of models of the processes underlying highly censored data [44]. Bayesian computational methods can thus be used as a flexible and powerful way to perform inference on unobserved parameters. Software such as openBUGS [53] and JAGS [54] are making the use of MCMC algorithms for model fitting accessible to non-specialists.

Parameterization becomes more difficult for large-scale simulation models like ABMs not only because ABMs present many more parameters to be fitted but also because they make it harder to derive an explicit likelihood function making impossible the use of MCMC in Bayesian computational methods or maximum likelihood estimation methods. One promising techniques that does not require full, explicit likelihood functions, and that is used in statistical ecology and DNA sequencing, is one potential solution: sequential importance sampling [55]. Sequential importance sampling, particle filtering or the sequential Monte Carlo method can be performed using the R package POMP [56].

### Implications for CCPV protocol

Reporting the combination of data used for parameterization would allow model users to evaluate the reliability of the models, reduce the risk of model overfitting and allow assessing the adequacy of the parameter for a specific setting (Table 3 “model parameterization”). Sensitivity and uncertainty analysis are other ways to evaluate the influence of individual parameters and their uncertainty range on model predictions (Table 2). They can be used to direct data collection efforts and should ideally be reported (Table 3 “sensitivity and uncertainty analysis”).