A Mixed-Methods Assessment of Understanding (AoU) Tool for HIV Vaccine Trials: Results From a Comparative Study in South Africa and Uganda Sabrina
1 Welsh ,
Graham
2 Lindegger ,
Michael
2 Quayle ,
3 Mark ,
4 Wallace ,
4 Bekker ,
4 Roux ,
5 Bahemuka ,
Daniella Melissa Linda-Gail Surita Ubaldo Frances Priddy1, Pat Fast1, Sagri Singh7, Judie Mbogua1, Prince Bahati1, Bonnie Bender1
Eugene
5 Ruzagira ,
Tsedal
6 Mebrahtu ,
1. International AIDS Vaccine Initiative, New York, NY, USA, 2. School of Psychology, University of KwaZulu-Natal, Pietermaritzburg, South Africa, 3. Paediatric AIDS Treatment for Africa/ Department of Psychology, University of Cape Town, Cape Town, South Africa, 4. Desmond Tutu HIV Foundation, Cape Town, South Africa, 5. MRC/UVRI Uganda Research Unit on AIDS, Masaka, Uganda, 6. US Military HIV Research Program, Henry M. Jackson Foundation, Bethesda, MD, USA, 7. GHAR Consulting Inc, New York, NY, USA
ABSTRACT Closed-ended methods, such as true/false tests, for assessment of understanding (AoU) may not measure true understanding of important trial concepts. For HIV vaccine trials, complex concepts require more comprehensive assessment methods. Eighty men and women were enrolled at two clinical research centers in South Africa and Uganda, which were recruiting for phase I HIV vaccine trials. A within-subjects, repeated measures design was used. Consenting volunteers were given three AoUs with different formats: a true/false test, a narrative, and scenarios. Interviews were recorded and transcribed. Volunteer feedback on the AoUs was collected with a questionnaire. Focus groups with study staff investigated experiences implementing the different formats. AoU transcripts were scored by one inexperienced and two experienced raters using standardized criteria, but poor inter-rater reliability analysis resulted in only scores from the experienced raters being retained. Performance was compared using correlations and Friedman's analysis of variance by ranks. Narrative and scenario scores were well correlated with each-other, but not with the true/false measure. The true/false and the narrative scores differed significantly, as did the true/false and scenario scores. The true/false test appeared to overestimate understanding, compared to the qualitative measures. Results from the volunteer interviews and staff focus groups showed that the scenario format was preferred because of volunteer comfort level and increased dialogue about the trial between volunteers and staff. The results suggest that qualitative methods may be better suited to assessing volunteer understanding in clinical trial settings, particularly for concepts that are considered essential for genuine informed consent. Given that the inter-rater reliability showed significant differences in how scorers were evaluating the AoU responses, it seems prudent to incorporate more training for individuals who administer and score AoUs in order to increase consistency and to verify staff comprehension of the concepts. An AoU toolkit is being developed that will incorporate findings from this study and additional experience using a mixed-method tool in recent HIV vaccine clinical trials.
Photo courtesy of the Medical Research Council/Uganda Virus Research Institute, Uganda Research Unit on AIDS
• Interviews were audio-recorded and scored by independent evaluators from each research center, IAVI, and HAVEG. Volunteer feedback on the AoU tools was collected using a questionnaire which asked participants which AoU they preferred and why. Study staff were asked to participate in focus groups that discussed staff preferences, overall feasibility, ease of use of administering the three AoUs, and challenges that were encountered.
• The inexperienced rater did not agree with the experienced raters, but the experienced raters had adequate agreement with each other. On the basis of the agreement between the experienced raters, the ratings from the inexperienced raters were dropped in this analysis. The disagreement between different raters may be due to the varying levels of training and previous experience scoring qualitative AoUs.
• Performance was compared using correlations and Friedman's analysis of variance by ranks.
RESULTS CONCLUSION
o The average age of volunteers was 25 (range 18-42). 52.5% and 82.5% of volunteers were female at MRC-Masaka and DTHF-Cape Town respectively.
Correlations (N=68)
• Volunteers were recruited from eligible adult potential volunteers for two HIV vaccine clinical trials at two clinical research centers: the Desmond Tutu HIV Foundation (DTHF) in Cape Town, South Africa, and the Medical Research Council/Uganda Virus Research Institute, Uganda Research Unit on AIDS in Masaka, Uganda. Participation in the AoU Comparative study was not required to enroll in the HIV vaccine trials. • Three AoU formats were compared: true/false (closed-ended questions), narrative (in which the volunteer describes to a staff member everything he/she understands about the trial), and a scenario vignette method (consisting of scenarios with probing questions for each concept.) Scenario 1: Administrator: please read the following scenario out loud to volunteer: • Four critical concepts were selected during a Mr. Smith decides that he will enroll in this vaccine trial. So, he starts attending information ranking exercise performed prior to the AoU sessions/discussion groups. On the way to attend one of his sessions at the vaccine trial site, he thinks to himself, “Once I am in the vaccine trial, I won’t need to worry about using protection study in which clinical research center staff, during sex anymore”. If Mr. Smith told you this what would you say to him? community advisory boards, and IAVI staff Score: 1= correct, 0 = wrong were asked to rank informed consent Concept 1: FALSE SENSE OF SECURITY 1. Some volunteers may not be getting study vaccine; they concepts in order of complexity and may be getting placebo and will not be protected against HIV infection importance to volunteer safety and personal If volunteer does not mention the above, use the following agency. question to get information Do all volunteers in the study get the same injection? What injections do they get? What kind of protection against HIV do you think these injections provide? 2. The study vaccine may not protect volunteers against HIV infection If volunteer does not mention the above, use the following question to get information What kind of protection against HIV do you think this study vaccine provides? 3. Volunteers must not increase their risk behavior If volunteer does not mention the above, use the following question to get information What should volunteers be doing to help them not get HIV?
Figure 1. Example of a scenario from the mixed-methods AoU describing the false sense of security that a trial volunteer might feel when receiving an HIV vaccine in the trial.
True/false
True/false Scenario Narrative
METHODS
• Education and performance on the narrative AoU showed modest correlation. Volunteers with less education scored lower on the narrative, and volunteers with more education scored higher.
• AoU transcripts were scored by one inexperienced and two experienced raters using standardized criteria, but poor inter-rater reliability analysis resulted in only scores from the experienced raters being retained.
Demographics:
Assessment of understanding (AoU) tools measure comprehension during the informed consent process and are often forced-choice questionnaires composed of true/false questions. In HIV vaccine clinical trials, where concepts are complex and often require a high level of understanding, assessing comprehension accurately is challenging. Evidence from previous studies (Lindegger et al 2006) indicates that forced-choice questions tended to overestimate understanding, while qualitative AoUs revealed more subtle levels of understanding. Three AoU methods were developed and tested in a collaboration between two clinical research centers in subSaharan Africa, the HIV/AIDS Vaccine Ethics Working Group (HAVEG), and the International AIDS Vaccine Initiative (IAVI).
• The analysis focused on the four critical concepts to allow for comparisons to be made between the three different AoUs. See Figure 1 for an example of a scenario of the “false sense of security” concept.
• The majority of volunteers scored high on all concepts in the true/false, but the same volunteers could not duplicate those high scores on the narrative and scenarios.
• Only the true/false AoU was used as part of the screening criteria for entry into the vaccine trials, since the other two AoU tools were being assessed for feasibility and utility. Volunteers completed all three AoUs at a single visit and performance was compared for each volunteer. Volunteers were randomized to receive either the narrative or scenarios after the true/false AoU to minimize bias. A subset of volunteers did not receive all three AoUs at the original study visit. For these volunteers, all three AoU tools were re-administered to limit any changes in knowledge level that might occur between administrations.
BACKGROUND
• The four critical concepts selected were: 1) false sense of security in HIV vaccine efficacy; 2) contraception; 3) false positive HIV test result; and 4) potential enhanced susceptibility to HIV infection if exposed through HIV risk behavior.
• Two clinical research centers enrolled a total of 80 volunteers into the AoU comparative study, which was conducted during the screening period of IAVI B002 and IAVI B003/IPCAVD-004/HVTN-091.
Scenario
Narrative
0.390 (p = .750)
-0.002 (p = .990)
0.390 (p = .750) -0.002 (p = .990)
0.591** (p