Although not all surveys conducted in complex emergencies are included in CE-DAT, we feel that some patterns can be identified when analyzing the more than 1000 surveys included in the database. One has to keep in mind though that this analysis can not necessarily be extrapolated to all existing surveys.
One of the main findings is that NGOs are the main source of surveys from complex emergency sites and are likely to remain so. Besides contributing the bulk of the data, there are important complementarities that the NGOs bring to the larger, more prestigious surveys undertaken by university groups or UN agencies.
First NGOs provide survey information at lower levels of resolution often camps, cities or districts while the UN surveys tend to cover large areas with the inherent problems of generalization in highly heterogeneous situations. Second, NGO surveys aim at assessing a local situation for needs and programming resources (e.g. beneficiaries, food vaccination doses) or for monitoring their impact. UN surveys tend to be large scale snap shots of a situation that serves as a point of reference rather than an operational tool (e.g. UNICEF's Multiple Indicator Cluster Surveys). Third, the NGO surveys provide essential contextual information, such as latest food arrivals or nearby fighting or raids which inform the analyses in volatile conditions. Most academic groups are usually stationed in the area temporarily and are located in less insecure areas. The UN monitors hostilities but as an inter-governmental authority has limitations with regard to presence in insecure parts and access to local information.
Finally, these surveys, whether undertaken by academics, UN or NGOs serve, in the first instance, practical and operational purposes related to priority setting for aid and programme input allocation. In this context therefore, it is important to distinguish quality needed for academic purposes from quality needed for operational purposes.
Although the highest level of scientific quality should be the objective of every study, the main purpose of conducting mortality and nutritional surveys is to provide reliable data rapidly for decision making for humanitarian aid. In this context, one could define a good survey as a survey that leads to a methodologically valid approximation rather than highly precise numbers.
Despite these strengths, the NGO surveys suffer from weaknesses. First consensus on appropriate thresholds and baselines to estimate excess deaths needs resolution, not only for the NGOs but for all the involved parties. Survey findings presented in a reference vacuum is essentially meaningless. Neither can the notion of severity ranking of the crises be useful without the concept of normality. NGOs are frequently the ones that ring alarm bells and alert the wider public to deteriorating situations. Guidelines to place their findings on a recognized scale would make their survey results useable for trend analyses, comparisons with other surveys and for evaluating severity.
Second, NGO survey reports should strengthen the peer review process without compromising on the rapidity of field-to-user turnaround time. The surveys undertaken by UN and research groups are built on sound scientific foundations but at a high cost in timeliness. Research groups typically do not make the survey results public until it has been accepted for scientific publication, easily a delay of 6 to 12 months. On the other hand, limited peer review, as is the case for most NGO surveys, affects both quality and credibility of their work – even if they are comparable to those done by the UN and academics. This being said, NGOs make their results available rapidly and within a useable timeframe. The optimal solution will be to recognize that the shelf life of survey findings in emergency situations is extremely short and a significantly rapid peer review mechanism will have to be found, so the results are not outdated for any practical decision making.
Finally, the paradigm of humanitarian response is shifting fast towards an evidence based response. Accountability and effectiveness of aid are strong motivating forces for donors towards their tax payers and NGOs towards their beneficiaries. NGOs have substantially strengthened their competences and technical skills in gathering credible data and providing evidence on the prevailing humanitarian situation. They are likely to remain the principle source of survey data. It is therefore, in the interests of the humanitarian community and the beneficiaries to further strengthen their capacities.