Panel survey
In the first and second waves, we recruited a nonprobability sample of 15,000 Japanese adults through a survey company, Rakuten Insight Ltd., a subsidiary of Rakuten Group, Inc. Respondents were solicited by Rakuten Insight and were rewarded with a certain number of points that could be used for shopping in an e-commerce market, Rakuten Ichiba, offered by Rakuten Group.
1 The first wave took place from February 17, 2021, to March 4, 2021, and the median response time was 7 minutes. The second wave took place from March 7, 2022, to March 22, and the median response time was 8 minutes. Out of the 15,000 respondents in the first wave, 9,668 also participated in the second wave, and this group formed our final sample.
In both waves, we asked respondents about their confidence in vaccine safety and the vaccine licensing authority’s process and about their recognition of the importance of vaccination. In both waves, we also asked about vaccine hesitancy, namely, whether they had previously taken vaccines other than the COVID-19 vaccines, whether they had previously postponed vaccines suggested by doctors, or whether they had previously refused vaccines, following categorization by MacDonald et al. [
41]. In the second wave, we also asked about their level of confidence in science. We followed Eichengreen et al. to create questions on confidence in science [
36].
In the second wave, we asked whether the participants had taken the first and second doses of the COVID-19 vaccine, whether they had already taken a third dose of the vaccine and if not, whether they wanted to do so, and whether they had experienced “COVID arm” symptoms. We also asked about their confidence in science and scientists in general.
In both waves, we asked about the following demographic, socioeconomic, and political characteristics: age, gender, marital status, number of children, gender and relative age of siblings, prefecture of residence, working status, highest educational degree, personal income, household income, party support, self-perceived degree of right-leaning political beliefs, degree of dissatisfaction with current politics, and self-perceived social status. In the second wave, we also asked whether the participants had any chronic diseases. Section SA
6 of the supplemental appendix presents an English translation of our questionnaire. To estimate the effects of “COVID arm” symptoms, we weighted our sample by sex, age, and employment status background pursuant to the composition of the latest population census in 2020.
2
A natural experiment conditional on background characteristics and prior beliefs
After studying the data we collected, we found that the occurrence of “COVID arm” symptoms was associated with certain background characteristics, as presented in Fig. A
2 in Section SA
2 of the supplemental appendix. Moreover, Fig. A
2 also demonstrates that experiencing “COVID arm” symptoms was also associated with prior general confidence in vaccine safety, prior confidence in the vaccine licensing authority, and previous behaviors of vaccine hesitancy. Therefore, we need to balance all factors, including demographic, socioeconomic, and political background characteristics, prior general confidence in vaccine safety, prior confidence in the vaccine licensing authority, and vaccine hesitancy, as described in
Balancing covariates section.
In summary, our data indicated that “COVID arm” symptoms and beliefs about vaccine safety, beliefs about the vaccine licensing authority in Japan, and beliefs about science in general were conditionally independent within individuals between the two waves given the background characteristics, prior general confidence in vaccine safety, prior confidence in the vaccine licensing authority, and prior vaccine hesitancy. Therefore, once controlling for background characteristics, prior confidence in vaccine safety, prior confidence in the vaccine licensing authority, and prior vaccine hesitancy, we can treat “COVID arm” symptoms as a natural experiment that randomly dropped in participants [
42]. We use this natural experiment to identify a causal relationship between an unexpected and uncomfortable experience with vaccination through belief updates about the side effects of vaccination and confidence in vaccine safety.
Note that it is not a weakness of our design but rather a strength that our data include respondents who were heterogeneous in terms of prior general confidence in vaccine safety, prior confidence in the vaccine licensing authority, and prior vaccine hesitancy. Under our panel design, we can trace both individuals who were confident in vaccination before experiencing “COVID arm” symptoms and those who were skeptical about it. We observed what happened to not only those who were confident in vaccination but also those who were skeptical about vaccination when experiencing “COVID arm” symptoms.
Identification and estimation strategy
Consider potential outcome
\(Y_{v^{l}, i}\left( D_{i}\right)\) regarding confidence in vaccination, where
\(D_{i}\) denotes whether the person experienced “COVID arm” symptoms such that
$$\begin{aligned} D_{i}=\left\{ \begin{array}{lc} 1 &{}\ \text {if respondent}\ i\ \text{experienced}\ \mathrm{``}\text{COVID arm''}\ \text{symptoms},\\ 0 &{}\ \text {otherwise}.\\ \end{array} \right. \end{aligned}$$
(1)
For
\(Y_{v^{l}, i}\left( D_{i}\right)\),
\(v^l\) denotes
\(v^{1}:\)
whether respondent i considered vaccination in general to be safe,
\(v^{2}:\)
whether respondent i was confident in the vaccine licensing process by the Japanese authority, the Ministry of Health, Labour and Welfare,
\(v^{3}:\)
whether respondent i acknowledged the importance of vaccination in general,
\(v^{4}:\)
whether respondent i took a second dose of the COVID vaccine,
\(v^{5}:\)
whether respondent i took or wanted to take a third dose of the COVID vaccine,
such that
$$\begin{aligned} Y_{v^{l}, i}\left( D_{i}\right) =\left\{ \begin{array}{lc} &{}1 \quad \text {if respondent}\ i\ \text{answers}\ \mathrm{``}\text{yes''},\\ &{}0 \quad \text {otherwise,}\\ \end{array} \right. \end{aligned}$$
(2)
for
\(v^{1},..., v^{5}\). Having taken a second dose (
\(v^{4}\)) and having taken or wanting to take a third dose (
\(v^{5}\)) are used to measure posterior vaccine hesitancy after taking the first dose.
As a supplementary survey, we asked the respondents about their confidence in science in general in the second wave such that
$$\begin{aligned} Y_{s, i}\left( D_{i}\right) =\left\{ \begin{array}{ll} &{}1 \quad \text {if respondent}\ i\ \text {was strongly confident in science},\\ &{}2 \quad \text {if respondent}\ i\ \text {was moderately confident in science},\\ &{}3 \quad \text {if respondent}\ i\ \text {was not very confident in science},\\ &{}4 \quad \text {if respondent}\ i\ \text {was not confident in science at all.} \end{array} \right. \end{aligned}$$
(3)
Then, we obtained three outcome variables such that
$$\begin{aligned} Y_{s^{1}, i}\left( D_{i}\right)= & {} \left\{ \begin{array}{ll} 1 &{} \text {if} \ \ Y_{s, i}\left( D_{i}\right) =1,\\ 0 &{} \text {otherwise}, \end{array} \right. \nonumber \\ Y_{s^{2}, i}\left( D_{i}\right)= & {} \left\{ \begin{array}{ll} 1 &{} \text {if} \ \ Y_{s, i}\left( D_{i}\right) \le 2,\\ 0 &{} \text {otherwise}, \end{array} \right. \nonumber \\ Y_{s^{3}, i}\left( D_{i}\right)= & {} \left\{ \begin{array}{ll} 1 &{} \text {if} \ \ Y_{s, i}\left( D_{i}\right) \le 3,\\ 0 &{} \text {otherwise}. \end{array} \right. \end{aligned}$$
(4)
Thus, the value of our interest is the expected difference in marginal means between having experienced “COVID arm” symptoms,
\(D_{i}=1\) and not having experienced such symptoms,
\(D_{i}=0\) such that
$$\begin{aligned} \tau \left( \varvec{x}\right) =E\left[ Y_{j, i}\left( 1\right) -Y_{j, i}\left( 0\right) |\varvec{X}_{i}=\varvec{x}\right] , \end{aligned}$$
(5)
where
\(j \in \{v^{1}, v^{2}, v^{3}, v^{4}, v^{5}, s^{1}, s^{2}, s^{3}\}\) and
\(\varvec{X}_{i}\) denotes the background characteristics, prior general confidence in vaccine safety, prior confidence in the vaccine licensing authority, and prior vaccine hesitancy of respondent
i.
As discussed in
A natural experiment conditional on background characteristics and prior beliefs section, “COVID arm” symptoms
\(D_{i}\) and confidence in vaccine safety, the vaccine licensing authority in Japan, and science
\(Y_{j, i}\) satisfy the unconfoundedness assumption conditional on background characteristics, prior general confidence in vaccine safety, prior confidence in the vaccine licensing authority, and prior vaccine hesitancy
\(\varvec{X}_{i}\) such that
$$\begin{aligned} D_{i} \perp \!\!\!\perp \left[ Y_{j, i}\left( 0\right) , Y_{j, i}\left( 1\right) \right] \big {|}\varvec{X}_{i}. \end{aligned}$$
Therefore, we identified Eq. (
5) as a causal effect of “COVID arm” symptoms
\(D_{i}\) on posterior general confidence in vaccine safety (
\(Y_{v^{1}, i}\)), posterior confidence in the vaccine licensing authority (
\(Y_{v^{2}, i}\)), posterior acknowledgment of the importance of vaccination (
\(Y_{v^{3}, i}\)), posterior vaccine hesitancy, (
\(Y_{v^{4}, i}\) and
\(Y_{v^{5}, i}\)), and posterior confidence in science (
\(Y_{s^{1}, i}\),
\(Y_{s^{2}, i}\), and
\(Y_{s^{3}, i}\)), given
\(\varvec{X}_{i}\).
We wanted to obtain the average treatment effect characterized by Eq. (
5) as the augmented inverse probability weighting (AIPW) with the double/debiased supervised machine learning algorithm [
43‐
46]. All nuisance functions are estimated by a causal forest algorithm [
47]. We conditioned all characteristics, including prior confidence in vaccine safety, prior confidence in the vaccine licensing authority, prior acknowledgment of the importance of vaccination, and prior vaccine hesitancy.
Before analyzing, we weighted our nonprobability sample by demographic characteristics pursuant to the latest population census in 2020, as mentioned above. Then, to control for confounders, the AIPW estimation with supervised machine learning has an obvious advantage regarding robustness against mis-specifications of the model for estimation. Traditional estimation methods such as OLS potentially depend on the specification of the estimation model. A slight misspecification in modeling might result in a substantial bias in estimation. Our approach allows us to semiparametrically estimate the average effect with asymptotical guarantees of debiasedness.
Note that this approach does not require parametric assumptions on the outcome distribution. If we have correct knowledge about the distribution, we may obtain more efficient estimators. However, in our design of the experiment, such knowledge is not available. Thus, we decided that AIPW estimation with supervised machine learning is the most relevant method compared to other possible candidate estimation methods.
In reality, experiencing “COVID arm” symptoms can be associated with prior confidence in vaccine safety, prior confidence in the vaccine licensing authority, prior acknowledgment of the importance of vaccination, and prior vaccine hesitancy, as presented in Table A
2 in Section SA
2 of the supplemental appendix. We control for all such variables in the AIPW estimation.
Finally, we regressed the AIPW score function on
\(\varvec{X}_{i}\) by OLS to obtain the best linear projection of
\(E\left[ \tau \left( \varvec{x}\right) \right]\). All algorithms are included in the
grf package for R [
48].
By balancing the sample with AIPW, we can observe the effects of belief updates due to “COVID arm” symptoms on both those who were confident in vaccination and those who were skeptical about vaccination before experiencing “COVID arm” symptoms. With our panel design, we can estimate the average treatment effects of “COVID arm” symptoms across the groups.