Narrative measure
Assessment of Understanding for Informed Consent in HIV Vaccine Trials Graham Lindegger, Michael Quayle, Catherine Slack HIV/AIDS Vaccine Ethics Group, University of KwaZulu-Natal, South Africa
Sagri Singh, Sabrina Welsh, Pat Fast International AIDS Vaccine Initiative, New York. ABSTRACT
EMPIRICAL INVESTIGATION/S
Concept 1: False sense of security
AIM: To compare methods of assessment of understanding of critical concepts in HIV vaccine trials in prospective trial
There were few significant correlations between True/False Test and Scenario/Narrative measures, suggesting that volunteers who did well one the one assessment measure did not do well on the other.
participants.
False tests tending to yield significantly higher scores than Scenario and Narrative measures, suggesting over-estimation of STUDY DESIGN: Two comparative studies were conducted using a repeated measures, within-subject design: each
trial, I won’t need to worry about practicing this safe sex stuff anymore.” What do True/False tests tended to yield significantly higher scores than Scenario and Narrative measures, suggesting over-estimation of understanding on True/False Test Inter-rater reliability between experienced and inexperienced scorers of qualitative assessment methods showed significant variance, showing the importance of training and experience in the use of qualitative
STUDY ONE: Conducted in South Africa and Zambia with participants in a hypothetical HIV vaccine trial. (N=36 ) STUDY TWO: Conducted in South Africa and Uganda with potential participants in an HIV vaccine trial.
assessment tools.
True/False Test: Forced choice questionnaire
Rigorous ethical standards are an integral part of HIV vaccine trials
Scenario measure: presentation of real life scenarios related to particular concepts, requiring the response of the
Informed consent, as a genuinely ethical, rather than legalistic process, is an essential ethical requirement1, 2, 3 Informed consent requires that potential participants understand various aspects of HIV vaccine trials in order to make good decisions about trial participation
THE CHALLENGE
STUDY 1
participant
False Sense of Security
Narrative measure: requesting participants to imagine telling a friend about trial participation, with assessors probing for narrative description of concepts
But there is controversy about how best to measure the level of understanding of HIV vaccine trial concepts1, 2
Study 1: compared True/False test and Scenario measure Study 2: compared True/False test and Scenario measure and Narrative measure
Need for Contraception
Risk of VISP
STUDY 2 Product Safety
Overall Understanding
N
36
36
36
36
36
Mean
.80
.84
.73
.78
.78
SD
.26
.23
.24
.24
.15
Table 1: Descriptive statistics for True/False Test of understanding
Measure of
But it is necessary for participants to adequately understand these concepts in order to ethically participate in HIV vaccine trials.
SELECTION OF CONCEPTS: An analysis was conducted by various HIV vaccine trial stakeholders in order to
False Sense of Security
determine the likely ethical consequences of not understanding each concept in HIV vaccine trials. On this basis the four
Difficult concepts include (amongst others):
concepts with the most serious ethical consequences of non-understanding were selected:
HIV vaccines may produce false positives on HIV testing (VISP) Participants are randomized to trial groups, and researchers and participants are blind as to which group participants belong to
False sense of security: danger of thinking the study uses a proven vaccine rather than research; therapeutic misconception
(double blind)
Risk of VISP
Need for Contraception
Product Safety
Overall Understanding
N
35
35
35
35
35
Mean
.56
.40
.60
.42
.50
SD .36 .33 .31 .23 .21 Table 2: Descriptive statistics for Scenario measures of understanding
Vaccine-induced sero-positivity (VISP): false positive results on routine HIV tests
HIV vaccines may enhance susceptibility to infection if exposed to the HI virus CONSENT1,2
Traditionally by True/False (forced-choice) questionnaires Easy to implement and score; time saving; require little counsellor skill
Risk of pregnancy/Need for contraception: unknown effect of vaccine on unborn baby Enhanced susceptibility/Product Safety: the vaccine cannot infect participants with HIV, but may enhance susceptibility to infection if exposed to the HI virus.
Concept False sense of security Risk of VISP Need for contraception Product safety Overall measures
Correlation .377* .228 .256 .393* .617**
May measure short term recall rather than understanding May be culturally foreign in forced-choice format
T False sense of security
Self-rated understanding of concepts: participants rate their personal understanding on each concept: but questionable validity and reliability
df P
95% confidence interval of the difference lower upper bound bound
-3.862 34 < .0005** -.356
-.111
Risk of VISP -7.210 34 < .0005** -.562
-.315
Need for contracep-2.303 34 .028* -.251 -.016 tion Product safe-8.007 34 < .0005** -.436 -.259 ty Total 34 < .0005** -.345 -.231 10.219 Table 4: Paired Samples t-tests between True/False Test and Scenario measure
True life scenarios related to concepts: Less likely to be based on short term recall or rote learning More likely to assess true understanding Likely to explore understanding of personal implications of each concept Likely to be more culturally congruent, especially in an African context
Risk of VISP
Need for contraception
Product safety
of the problem.” What do you think about her reaction?
Concept 3: Need for contraception
Core elements
Six months later, while still in the trial, Mrs Dlamini and her husband want to have
Participants should understand the following elements: (one point for each
another child. She reports that she is thinking of stopping her
aspect reported correctly)
Overall understanding
If a participant becomes HIV infected, the vaccine could not have
contraception. What do you think of this?
True/false test
Scenario measure
Mean
0.993
0.956
0.963
0.927
0.960
SD
0.061
0.167
0.132
0.199
03.091
getting HIV from high-risk behaviour
Mean
0.97096
0.9044
0.9314
0.6814
0.8702
Six months after joining the trial, Mrs Jones tests HIV-infected. She says to the
SD
0.13792
0.218
0.13578
0.17214
0.11522
counsellor: “I knew this vaccine would infect me with HIV”. She is very angry.
of HIV infection because of the vaccine.
Then she adds: “Even if the vaccine didn’t give me the HIV, it is part of the prob-
Faulty information
lem”. What do you think about her reaction?
Any claim that the vaccine can directly cause HIV infection means that the
Narrative measure Mean SD
0.9495
0.9191
0.1653
0.2678
0.9403
0.7181
0.18049
0.8797
0.27205
Concept 4: Vaccine cannot cause infection but might increase the risk of
If a participant becomes HIV infected, they must have been exposed to the HI virus, primarily through high risk behavior e.g. unprotected sex.
0.13544
If a participant is exposed to the HI virus, there may be an enhanced risk
participant scores zero for this whole concept, even if they get some other
Table 5: Descriptive statistics for different measures of understanding
elements correct.
Box 2: Example of administration and scoring of scenarios
Box 1: Scenarios based on concepts to be understood Correlations between True/False Test and Narrative measures
Concept
Correlations between True/False Test and Scenario measures
False sense of security
-.026
Risk of VISP
-.118
-.038 -.081
Need for contraception
.135
-.095
.069
.071
Open-ended qualitative assessment methods (Scenarios and Narratives) probably provide a more valid and reliable assessment of levels of understanding of critical trial
-.002
concepts
OVERALL SCALES
.039
CONCLUSIONS The findings of these studies confirm those of Lindegger (2006) regarding the difference in measures of understanding on different assessment instruments.
Given the greater level of skill and expertise required to use and score these methods, it is recommended that these tools are only used for assessing the most critical con-
Table 6: Correlations between True/False Test and Scenario/ Narrative measure
cepts in trials, such as the four used in this study Counsellors using qualitative tools require better levels of understanding of concepts themselves, and more training and expertise in the use of the Friedman Χ2
df
P instruments
False sense of security Risk of VISP
4.710 5.851
2 2
.095 This study suggests that research teams are taking seriously ethical recommendations to engage stakeholders in the development of strategies to test understanding2and to
.054
develop interpersonal skills to strengthen the consent process3. Need for contraception
2.147
2
.342
Product safety
38.138
2
.0005
Total
35.395
2
.0005
Table 7: Friedman’s non-parametric comparisons of True/False Test, Scenario and Narrative measures
REFERENCES 1. Lindegger,G. & Richter, L. (2000). HIV vaccine trials: critical issues in informed consent, South African Journal of Science, 96, 313-317 2.
May be time consuming and complex to score
angry. Then she adds: “Even if the vaccine didn’t give me the HIV, it is part
directly infected them.
Product safety
Table 3: Correlations between True/False Test and Scenario measure
But these may assess rote learning of technical concepts rather than understanding
False sense of security
understanding
Many aspects of trials are difficult to understand
Lindegger,G. , Milford, C., Slack, C., Quayle, M., Xaba, X., Vardas, E J. (2006). Beyond the checklist: assessing understanding for HIV vaccine trial participation in South Africa. Acquir Immune Defic Syndr., 43, 560-6.
3. Medical Research Council [MRC] (2003). Guidelines on ethics for medical research, Book 5: HIV preventive vaccine research. Retrieved May 15, 2009, from http://www.mrc.ac.za/ethics/ ethicsbook5.pdf
Narrative description of concepts in lay termin0logy: potential participants describe to an imaginary friend what each concept is
4. UNAIDS/AVAC (2011). Good participatory practice: Guidelines for biomedical HIV prevention trials. Geneva: UNAIDS. Retrieved April 21, 2012, from http://www.unaids.org/en/ media/unaids/contentassets/documents/unaidspublication/2011/JC1853_GPP_Guidelines_2011_en.pdf
about: Same dis/advantages as scenarios
because of often being sick. The doctor advises that she has an HIV test. The re-
TOOLS: Compared three methods to assess understanding of HIV vaccine concepts:
ETHICS OF HIV VACCINE TRIALS
ASSESSMENT OF UNDERSTANDING FOR INFORMED
the counselor: “I knew this vaccine would infect me with HIV”. She is very
sults come back positive. What do you think about this?
BACKGROUND TO STUDY
The HIV vaccine cannot directly cause HIV infection in participants;
Six months after joining the trial, Mrs Jone tests HIV-infected. She says to
While participating in an HIV vaccine trial, Mrs Jones goes to the clinic
Participants and counsellors tended to most prefer scenario based methods of assessment
Description
exposed to the HI virus. Scenario presented to participants Concept 2: Risk of false positive HIV test
Qualitative methods did not take significantly longer to administer than forced choice questionnaires
enhanced susceptibility
however, there is the possibility of enhanced susceptibility to infection if
you think about what she says to herself ?
participant completed 2 or 3 measures of understanding, with order of presentation varied to control for practice effects. understanding, and should be selected in screening prospective participants for HIV vaccine trials.
attending information sessions and discussion groups. On the way to attend one of her sessions at the vaccine trial site, she thinks to herself, “Once I am in the vaccine
There were significant differences between scores on True/False tests on the one hand, and Scenario/ Narrative measures on the other hand
understanding on True/False Test. Findings suggest that qualitative measures provide more valid and reliable measures of
Product safety: No risk of direct infection, but small chance of
Mrs Dlamini decides that she will enrol in the vaccine trial. So, she starts
Scores on various measures of understanding were compared using Friedman tests, correlations and paired samples t-tests
Levels of understanding of HIV vaccine trial concepts on three measurement instruments were compared. There were significant differences between scores on True/False tests on the one hand, and Scenario and Narrative measures on the other hand, with True/
FINDINGS OF THE TWO STUDIES:
Figure 1: Methods of assessing understanding for HIV vaccine trials
Sagri Singh, Sabrina Welsh, Pat Fast International AIDS Vaccine Initiative, New York. ABSTRACT
EMPIRICAL INVESTIGATION/S
Concept 1: False sense of security
AIM: To compare methods of assessment of understanding of critical concepts in HIV vaccine trials in prospective trial
There were few significant correlations between True/False Test and Scenario/Narrative measures, suggesting that volunteers who did well one the one assessment measure did not do well on the other.
participants.
False tests tending to yield significantly higher scores than Scenario and Narrative measures, suggesting over-estimation of STUDY DESIGN: Two comparative studies were conducted using a repeated measures, within-subject design: each
trial, I won’t need to worry about practicing this safe sex stuff anymore.” What do True/False tests tended to yield significantly higher scores than Scenario and Narrative measures, suggesting over-estimation of understanding on True/False Test Inter-rater reliability between experienced and inexperienced scorers of qualitative assessment methods showed significant variance, showing the importance of training and experience in the use of qualitative
STUDY ONE: Conducted in South Africa and Zambia with participants in a hypothetical HIV vaccine trial. (N=36 ) STUDY TWO: Conducted in South Africa and Uganda with potential participants in an HIV vaccine trial.
assessment tools.
True/False Test: Forced choice questionnaire
Rigorous ethical standards are an integral part of HIV vaccine trials
Scenario measure: presentation of real life scenarios related to particular concepts, requiring the response of the
Informed consent, as a genuinely ethical, rather than legalistic process, is an essential ethical requirement1, 2, 3 Informed consent requires that potential participants understand various aspects of HIV vaccine trials in order to make good decisions about trial participation
THE CHALLENGE
STUDY 1
participant
False Sense of Security
Narrative measure: requesting participants to imagine telling a friend about trial participation, with assessors probing for narrative description of concepts
But there is controversy about how best to measure the level of understanding of HIV vaccine trial concepts1, 2
Study 1: compared True/False test and Scenario measure Study 2: compared True/False test and Scenario measure and Narrative measure
Need for Contraception
Risk of VISP
STUDY 2 Product Safety
Overall Understanding
N
36
36
36
36
36
Mean
.80
.84
.73
.78
.78
SD
.26
.23
.24
.24
.15
Table 1: Descriptive statistics for True/False Test of understanding
Measure of
But it is necessary for participants to adequately understand these concepts in order to ethically participate in HIV vaccine trials.
SELECTION OF CONCEPTS: An analysis was conducted by various HIV vaccine trial stakeholders in order to
False Sense of Security
determine the likely ethical consequences of not understanding each concept in HIV vaccine trials. On this basis the four
Difficult concepts include (amongst others):
concepts with the most serious ethical consequences of non-understanding were selected:
HIV vaccines may produce false positives on HIV testing (VISP) Participants are randomized to trial groups, and researchers and participants are blind as to which group participants belong to
False sense of security: danger of thinking the study uses a proven vaccine rather than research; therapeutic misconception
(double blind)
Risk of VISP
Need for Contraception
Product Safety
Overall Understanding
N
35
35
35
35
35
Mean
.56
.40
.60
.42
.50
SD .36 .33 .31 .23 .21 Table 2: Descriptive statistics for Scenario measures of understanding
Vaccine-induced sero-positivity (VISP): false positive results on routine HIV tests
HIV vaccines may enhance susceptibility to infection if exposed to the HI virus CONSENT1,2
Traditionally by True/False (forced-choice) questionnaires Easy to implement and score; time saving; require little counsellor skill
Risk of pregnancy/Need for contraception: unknown effect of vaccine on unborn baby Enhanced susceptibility/Product Safety: the vaccine cannot infect participants with HIV, but may enhance susceptibility to infection if exposed to the HI virus.
Concept False sense of security Risk of VISP Need for contraception Product safety Overall measures
Correlation .377* .228 .256 .393* .617**
May measure short term recall rather than understanding May be culturally foreign in forced-choice format
T False sense of security
Self-rated understanding of concepts: participants rate their personal understanding on each concept: but questionable validity and reliability
df P
95% confidence interval of the difference lower upper bound bound
-3.862 34 < .0005** -.356
-.111
Risk of VISP -7.210 34 < .0005** -.562
-.315
Need for contracep-2.303 34 .028* -.251 -.016 tion Product safe-8.007 34 < .0005** -.436 -.259 ty Total 34 < .0005** -.345 -.231 10.219 Table 4: Paired Samples t-tests between True/False Test and Scenario measure
True life scenarios related to concepts: Less likely to be based on short term recall or rote learning More likely to assess true understanding Likely to explore understanding of personal implications of each concept Likely to be more culturally congruent, especially in an African context
Risk of VISP
Need for contraception
Product safety
of the problem.” What do you think about her reaction?
Concept 3: Need for contraception
Core elements
Six months later, while still in the trial, Mrs Dlamini and her husband want to have
Participants should understand the following elements: (one point for each
another child. She reports that she is thinking of stopping her
aspect reported correctly)
Overall understanding
If a participant becomes HIV infected, the vaccine could not have
contraception. What do you think of this?
True/false test
Scenario measure
Mean
0.993
0.956
0.963
0.927
0.960
SD
0.061
0.167
0.132
0.199
03.091
getting HIV from high-risk behaviour
Mean
0.97096
0.9044
0.9314
0.6814
0.8702
Six months after joining the trial, Mrs Jones tests HIV-infected. She says to the
SD
0.13792
0.218
0.13578
0.17214
0.11522
counsellor: “I knew this vaccine would infect me with HIV”. She is very angry.
of HIV infection because of the vaccine.
Then she adds: “Even if the vaccine didn’t give me the HIV, it is part of the prob-
Faulty information
lem”. What do you think about her reaction?
Any claim that the vaccine can directly cause HIV infection means that the
Narrative measure Mean SD
0.9495
0.9191
0.1653
0.2678
0.9403
0.7181
0.18049
0.8797
0.27205
Concept 4: Vaccine cannot cause infection but might increase the risk of
If a participant becomes HIV infected, they must have been exposed to the HI virus, primarily through high risk behavior e.g. unprotected sex.
0.13544
If a participant is exposed to the HI virus, there may be an enhanced risk
participant scores zero for this whole concept, even if they get some other
Table 5: Descriptive statistics for different measures of understanding
elements correct.
Box 2: Example of administration and scoring of scenarios
Box 1: Scenarios based on concepts to be understood Correlations between True/False Test and Narrative measures
Concept
Correlations between True/False Test and Scenario measures
False sense of security
-.026
Risk of VISP
-.118
-.038 -.081
Need for contraception
.135
-.095
.069
.071
Open-ended qualitative assessment methods (Scenarios and Narratives) probably provide a more valid and reliable assessment of levels of understanding of critical trial
-.002
concepts
OVERALL SCALES
.039
CONCLUSIONS The findings of these studies confirm those of Lindegger (2006) regarding the difference in measures of understanding on different assessment instruments.
Given the greater level of skill and expertise required to use and score these methods, it is recommended that these tools are only used for assessing the most critical con-
Table 6: Correlations between True/False Test and Scenario/ Narrative measure
cepts in trials, such as the four used in this study Counsellors using qualitative tools require better levels of understanding of concepts themselves, and more training and expertise in the use of the Friedman Χ2
df
P instruments
False sense of security Risk of VISP
4.710 5.851
2 2
.095 This study suggests that research teams are taking seriously ethical recommendations to engage stakeholders in the development of strategies to test understanding2and to
.054
develop interpersonal skills to strengthen the consent process3. Need for contraception
2.147
2
.342
Product safety
38.138
2
.0005
Total
35.395
2
.0005
Table 7: Friedman’s non-parametric comparisons of True/False Test, Scenario and Narrative measures
REFERENCES 1. Lindegger,G. & Richter, L. (2000). HIV vaccine trials: critical issues in informed consent, South African Journal of Science, 96, 313-317 2.
May be time consuming and complex to score
angry. Then she adds: “Even if the vaccine didn’t give me the HIV, it is part
directly infected them.
Product safety
Table 3: Correlations between True/False Test and Scenario measure
But these may assess rote learning of technical concepts rather than understanding
False sense of security
understanding
Many aspects of trials are difficult to understand
Lindegger,G. , Milford, C., Slack, C., Quayle, M., Xaba, X., Vardas, E J. (2006). Beyond the checklist: assessing understanding for HIV vaccine trial participation in South Africa. Acquir Immune Defic Syndr., 43, 560-6.
3. Medical Research Council [MRC] (2003). Guidelines on ethics for medical research, Book 5: HIV preventive vaccine research. Retrieved May 15, 2009, from http://www.mrc.ac.za/ethics/ ethicsbook5.pdf
Narrative description of concepts in lay termin0logy: potential participants describe to an imaginary friend what each concept is
4. UNAIDS/AVAC (2011). Good participatory practice: Guidelines for biomedical HIV prevention trials. Geneva: UNAIDS. Retrieved April 21, 2012, from http://www.unaids.org/en/ media/unaids/contentassets/documents/unaidspublication/2011/JC1853_GPP_Guidelines_2011_en.pdf
about: Same dis/advantages as scenarios
because of often being sick. The doctor advises that she has an HIV test. The re-
TOOLS: Compared three methods to assess understanding of HIV vaccine concepts:
ETHICS OF HIV VACCINE TRIALS
ASSESSMENT OF UNDERSTANDING FOR INFORMED
the counselor: “I knew this vaccine would infect me with HIV”. She is very
sults come back positive. What do you think about this?
BACKGROUND TO STUDY
The HIV vaccine cannot directly cause HIV infection in participants;
Six months after joining the trial, Mrs Jone tests HIV-infected. She says to
While participating in an HIV vaccine trial, Mrs Jones goes to the clinic
Participants and counsellors tended to most prefer scenario based methods of assessment
Description
exposed to the HI virus. Scenario presented to participants Concept 2: Risk of false positive HIV test
Qualitative methods did not take significantly longer to administer than forced choice questionnaires
enhanced susceptibility
however, there is the possibility of enhanced susceptibility to infection if
you think about what she says to herself ?
participant completed 2 or 3 measures of understanding, with order of presentation varied to control for practice effects. understanding, and should be selected in screening prospective participants for HIV vaccine trials.
attending information sessions and discussion groups. On the way to attend one of her sessions at the vaccine trial site, she thinks to herself, “Once I am in the vaccine
There were significant differences between scores on True/False tests on the one hand, and Scenario/ Narrative measures on the other hand
understanding on True/False Test. Findings suggest that qualitative measures provide more valid and reliable measures of
Product safety: No risk of direct infection, but small chance of
Mrs Dlamini decides that she will enrol in the vaccine trial. So, she starts
Scores on various measures of understanding were compared using Friedman tests, correlations and paired samples t-tests
Levels of understanding of HIV vaccine trial concepts on three measurement instruments were compared. There were significant differences between scores on True/False tests on the one hand, and Scenario and Narrative measures on the other hand, with True/
FINDINGS OF THE TWO STUDIES:
Figure 1: Methods of assessing understanding for HIV vaccine trials
