Background
Garenne and Faveau
In this paper, we report the results of a study designed specifically to examine the repeatability of the VA process. The study was run in parallel in Burkina Faso and Indonesia, to allow comparison between two very different settings, both of which involved repeated VA interviews concerning deaths among women of reproductive age. We chose to exclude consideration of repeatability of the interpretation stage of the process, by using the InterVA-M model
Our major aim was to assess the repeatability of the overall VA process under operational conditions, from interview to aggregated mortality patterns, in the two different settings.
Subsidiary aims were:
1. to assess the repeatability of the interview stage of the VA process (in terms of material gathered in VA interviews);
2. to assess the repeatability of cause of death determination at the individual level;
3. to assess the repeatability of cause-specific mortality fractions (CSMF) determined at the population level.
It should be noted that this study did not aim to arrive at conclusions, other than on repeatability, about cause of death patterns in either setting, and not explain differences in mortality patterns between the two countries, evaluate the clinical validity of VA, nor draw conclusions for health planning.
Methods
In both Burkina Faso and Indonesia, large scale community-based surveys of mortality among women of reproductive age, with a particular focus on pregnancy-related mortality, were undertaken in 2005–6 (February to May 2006 in Burkina Faso and December 2005 to June 2006 in Indonesia)
Conceptual framework for assessing the repeatability of verbal autopsy interviews
Conceptual framework for assessing the repeatability of verbal autopsy interviews.
The interview material thus consisted of individual sets of responses to the 75 possible InterVA-M indicators, and the repeatability of the interview stage of the overall process was assessed by calculating kappa statistics for each indicator, together with p-values assessing whether agreement between the original and follow-up interviews was significantly greater than that expected by chance.
The InterVA-M model was then run on all the original and follow-up interview material from both countries, to interpret likely pregnancy status and cause-of-death outcomes. The InterVA-M model generated, for each case, the most likely pregnancy status at the time of death (pregnant, delivered within 6 weeks or not pregnant within 6 weeks of death) with an associated likelihood. Then up to three likely alternative causes of death were generated, each with an associated likelihood. These likelihoods were used to ascribe fractional causes of death, as described previously
The individual pregnancy status and cause of death outputs were then combined into overall pregnancy status fractions and CSMFs for the population samples in Burkina Faso and Indonesia respectively, which were then compared in terms of magnitude and rank order.
Ethical approvals for the verbal autopsy studies in Burkina Faso and Indonesia, including follow-up interviews where needed, were granted by the Ministry of Health National Health Research Ethics Committee (Ouagadougou, Burkina Faso) and Centre MURAZ Institutional Review Board (Bobo-Dioulasso, Burkina Faso); by the Faculty of Public Health Research Ethics Committee at the University of Indonesia (Jakarta, Indonesia).
Results
A total of 207 VA interviews were successfully repeated, 91 in Burkina Faso and 116 in Indonesia. The basic characteristics of these cases (both the deceased women and the interview respondents) are summarised in Table
Background characteristics for verbal autopsy interviews in Burkina Faso (n = 91) and Indonesia (n = 116).
subject
characteristic
level
%
Burkina Faso
Indonesia
deceased woman
education
primary
2.2
62.9
secondary/higher
0
29.3
work for income
yes
45.1
37.9
place of death
health facility
51.7
45.7
water supply
piped
1.1
30.2
television
yes
2.2
50.0
respondent
sex
female
26.4
57.8
age
under 40 yrs
30.8
44.0
education
primary
8.8
62.1
secondary/higher
2.2
11.3
work for income
yes
90.1
62.1
Selected factors recorded at follow-up interviews, for women who had died and primary verbal autopsy interview respondents, are shown.
Kappa statistics were calculated for each InterVA-M indicator in each country to assess the repeatability of the verbal autopsy interview process. The distribution of kappa statistics for all measurable InterVA-M indicators in relation to their positive response rate in each country are shown in Figure
Repeatability of VA indicators in Burkina Faso. Details of repeatability for each verbal autopsy indicator from a series of 91 repeated interviews in Burkina Faso (by ascending κ values within each category).
Click here for file
Repeatability of VA indicators in Indonesia. Details of repeatability for each verbal autopsy indicator from a series of 116 repeated interviews in Burkina Faso (by ascending κ values within each category).
Click here for file
Distribution of kappa statistics versus positive response rate for indicators obtained from repeated verbal autopsy interviews in Burkina Faso (n = 91) and Indonesia (n = 116)
Distribution of kappa statistics versus positive response rate for indicators obtained from repeated verbal autopsy interviews in Burkina Faso (n = 91) and Indonesia (n = 116).
Since recall period is obviously a potentially important factor in the repeatability of VA interviews, mean kappa statistics were also calculated separately for original interviews falling before or after the median recall time, for each country. In Burkina Faso the median time to first interview was 26 months, and for 51 indicators represented in interviews on both sides of the median, the mean κ was 0.27 for interviews up to and including the median, compared with 0.17 for interviews after the median. Of the 51 indicators, 76% had a lower κ value for interviews beyond the median recall time. In Indonesia, the median time to first interview was 7 months, and for 60 indicators the mean κ was 0.44 for interviews up to and including the median, compared with 0.45 for interviews after the median time.
Table
Individual repeatability of pregnancy status and cause-of-death assessment from repeated verbal autopsy interviews in Burkina Faso (n = 91) and Indonesia (n = 116)
Burkina Faso
Indonesia
Likely cause of death
% of interviews leading to this likely cause/status
of which, % with cause/status found in both interviews
% of interviews leading to this likely cause/status
of which, % with cause/status found in both interviews
Abortion related death
2.7
5.8
1.6
10.8
Anaemia
5.2
0
5.0
14.9
Cancer
2.9
20.3
4.1
20.5
Cardiovascular disease
5.7
5.0
6.3
24.8
Diabetes
4.1
19.0
4.9
35.2
Ectopic pregnancy
0.8
0
2.3
10.4
HIV/AIDS related death
5.0
9.9
1.0
0
Haemorrhage
8.8
17.2
14.4
28.4
Indeterminate
1.2
100.0
2.8
100.0
Kidney disease
2.0
13.6
3.2
18.7
Liver disease
1.9
0
4.2
23.3
Malaria
24.1
24.1
8.3
30.2
Obstructed labour
2.1
6.6
2.3
12.1
Other maternal cause
0.8
0
2.8
23.9
Pregnancy-induced hypertension
3.5
0
6.0
11.5
Pregnancy-related infection
20.8
29.5
17.8
22.1
Respiratory disease
0.6
0
0.2
0
Ruptured uterus
1.1
0
1.3
0
Suicide
0.9
0
0.7
35.3
Tuberculosis (pulmonary)
5.7
10.5
10.8
27.0
OVERALL
100.0
18.0
100.0
25.0
Pregnancy status
not pregnant within 6 wks of death
6.7
66.0
8.0
82.2
pregnant within 6 wks of death
70.8
78.0
51.5
79.1
pregnant at death
22.7
34.0
40.5
90.1
OVERALL
100.0
67.2
100.0
85.0
Table
Overall cause-specific mortality fractions and ranked causes, together with pregnancy status assessments, from verbal autopsies in Burkina Faso (n = 91) and Indonesia (n = 116), in original and follow-up interviews
Burkina Faso
Indonesia
original interview
follow-up interview
original interview
follow-up interview
cause of death
%
rank
%
rank
%
rank
%
rank
Abortion related death
3.6
10
1.8
11
1.3
17
1.9
15
Anaemia
6.8
5
3.6
7
5.4
7
4.7
8
Cancer
2.2
13
3.6
7
2.9
11
5.2
7
Cardiovascular disease
8.4
3
3.0
9
7.0
6
5.5
5
Diabetes
4.1
9
4.1
5
4.5
9
5.3
6
Ectopic pregnancy
0.3
19
1.4
14
1.8
15
2.9
13
HIV/AIDS related death
5.4
7
4.6
5
0.3
20
1.8
16
Haemorrhage
8.2
4
9.4
3
17.2
1
11.5
2
Indeterminate
1.2
17
0
18
2.8
12
0.9
18
Kidney disease
2.4
12
1.5
13
3.3
10
3.0
12
Liver disease
1.6
15
2.1
10
5.2
8
3.3
11
Malaria
19.6
1
30.0
1
8.8
4
7.8
4
Non-pregnancy related infection
0
20
0
18
0
21
0.9
18
Obstructed labour
2.6
11
1.7
12
2.6
13
2.0
14
Other maternal cause
1.6
16
0
18
2.1
14
3.6
10
Pregnancy-induced hypertension
5.8
6
1.2
15
8.3
5
3.7
9
Pregnancy-related infection
18.5
2
23.2
2
11.8
2
24.2
1
Respiratory disease
0
20
1.2
17
0.3
19
0
21
Ruptured uterus
2.2
14
0
18
1.6
16
1.0
17
Suicide
0.6
18
1.2
15
1.0
18
0.4
20
Tuberculosis (pulmonary)
5.0
8
6.4
4
11.8
3
10.6
3
Pregnancy status
not pregnant within 6 wks of death
6.8
3
21.5
2
8.2
3
7.8
3
pregnant within 6 wks of death
70.9
1
67.0
1
51.4
1
51.6
1
pregnant at death
22.3
2
11.5
3
40.4
2
40.6
2
Table
Top five causes of death from original and follow-up interviews in series of verbal autopsies in Burkina Faso (n = 91) and Indonesia (n = 116).
Burkina Faso
Indonesia
original interview
follow-up interview
original interview
follow-up interview
rank
cause of death
CSMF %
cause of death
CSMF %
cause of death
CSMF %
cause of death
CSMF %
1
malaria
19.6
Malaria
30.0
haemorrhage
17.2
pregnancy-related infection
24.2
2
pregnancy-related infection
18.5
pregnancy-related infection
23.2
pregnancy-related infection
11.8
haemorrhage
11.5
3
cardio-vascular disease
8.4
Haemorrhage
9.4
tuberculosis (pulmonary)
11.8
tuberculosis (pulmonary)
10.6
4
haemorrhage
8.2
tuberculosis (pulmonary)
6.4
malaria
8.8
malaria
7.8
5
anaemia
6.8
HIV/AIDS related death
4.6
pregnancy-induced hypertension
8.3
cardio-vascular disease
5.5
total
61.5
73.6
57.9
59.6
Discussion
The concept of VA, from the interview, through interpretation, to population-level results, is a complex one. This study has investigated the repeatability of three different stages of this overall process, which is entirely novel, and capitalised on the use of the InterVA-M interpretative model in order to eliminate any subjective or inter-country variation in the process of interpreting the VA interview material.
Methodological issues
It is difficult to discuss these findings extensively in relation to other work, since very little attention has previously been given to the repeatability of VA. All of our interviews were conducted in real field conditions, typical of settings in which VA is an important tool, and where deaths in hospital are rare. We happened to use material relating to women of reproductive age, although it is likely that similar findings would apply to other population groups. While it is possible that we may have influenced the conduct of the VA process by, exceptionally, undertaking follow-up interviews, we believe this is unlikely. In all cases the interval between the original and follow-up interviews was between one and two years, which probably minimised any effect associated with recalling the previous interview, even where the same respondent was involved. On the other hand, the additional accumulation of recall time since the death itself may have influenced the way in which original events were remembered, and thus reduced repeatability.
VA respondents, recall and repeatability
In terms of the repeatability of the VA interview stage, it is clear that there were major differences between the interviews undertaken in Burkina Faso and Indonesia, with a markedly higher mean kappa statistic and proportion of indicators with non-chance agreement in Indonesia (Figure
When the individual-level pregnancy status and cause of death outputs were compared between the original and follow-up interviews, there was a disappointing lack of concordance, though this was perhaps not surprising given the extent of non-agreement between the interview material, particularly from Burkina Faso (Table
However, when the output data were considered at the population level, there was a much clearer sense of agreement between the original and follow-up material (Table
Implications of VA repeatability for health policy
Taking a public health perspective, Table
The relative levels of repeatability, both between the two country settings involved and between the different stages of the VA process, are interesting. The obvious differences between the two countries make clear that the whole VA process is context-dependent, since it might reasonably be inferred that in a context where repeatability is lower, then the intrinsic validity of one-time interview material would also be lower. The effects of non-repeatability at different stages of the VA process are also interesting – our results suggest that, given moderate repeatability within the original interviews, the repeatability of cause of death at the individual level is seriously compromised. However, when a reasonable group of individual VAs are taken as aggregate entities (around 100 per setting in these results) then our findings suggest that there is some recovery of repeatability, certainly to levels that appear to be acceptable in terms of generating aggregate data for health planning. The effects behind this are not easy to quantify, but it seems likely that some of the non-repeatability of details at the interview stage may have the effect of tipping the balance between possible causes at the individual level, but with many such differences then cancelling out on aggregating causes to the population level.
Why repeatability?
Repeatability is often not explicitly assessed for health measurement tools – so why is it important in relation to VA and what do the results tell us? The origins of VA – as a proxy source of cause of death data in the absence of medical certification – has sometimes led to confusion about its fundamental nature. While it is not generally argued that VA should be used as a direct replacement for medical certification at the individual level, this principle has not always been made explicit in VA work. Consequently, attempts to validate VA
We therefore offer these analyses of VA repeatability as a fresh viewpoint on the overall process, to shed some light on the practice of VA under realistic operational conditions and the value of the ensuing results. This partly follows from our previous consideration of the question "Who needs cause of death data?"
At least in contexts emphasising community-level cause-specific mortality, the findings of these repeatability analyses are encouraging. Although good repeatability does not guarantee good validity, at least it suggests that intrinsic validity is not compromised by random effects in the overall process. It also became clear that the longer recall periods associated with some of the Burkinabé interviews were detrimental to repeatability – and so presumably to validity. At the same time, overall repeatability was lower in Burkina Faso than in Indonesia, possibly because of different respondent profiles, which again emphasises the importance of considering VA material contextually, rather than simply in terms of standardised methods.
Conclusion
The overall process of VA, from interview to CSMF, has been shown to have good repeatability for two very different communities. However, VA outcomes were less repeatable at the individual level, and recall periods beyond two years compromised repeatability. Although repeatability does not demonstrate validity, it is a prerequisite, and so this study adds new understanding to the quest for reliable cause of death assessment in communities lacking routine medical certification of deaths.
Competing interests
The authors declare that they have no competing interests.
Authors' contributions
PB devised the study; all authors participated in field work; MO and SNQ coordinated work in Burkina Faso and Indonesia respectively and prepared datasets; PB & LD carried out analysis and drafting.