Direct and Indirect Effects—An Information Theoretic Perspective

We will be developing techniques for measuring the causal influence of $X\in \mathcal{X}$ upon $Y\in \mathcal{Y}$ in the presence of a mediating variable $Z\in \mathcal{Z}$ using the DAG $\mathcal{G}$ depicted in Figure 1. Without loss of generality, Z may represent a collection $(Z_1,Z_2,\cdots ,Z_k)\in {\mathcal{Z}}_1\times {\mathcal{Z}}_2\times \cdots \times {\mathcal{Z}}_k=\mathcal{Z}$ of all mediating variables. Define the parent sets of X, Z, and Y as $P{A}_X=U_X$, $P{A}_Z=\left\{X\right\}\cup U_Z$, and $P{A}_Y=\{X,Z\}\cup U_Y$. Dashed double headed arrows in Figure 1 are used to indicate unknown dependencies between $U_X\in {\mathcal{U}}_X$, $U_Y\in {\mathcal{U}}_Y$, and $U_Z\in {\mathcal{U}}_Z$ (including the possibility of $U_S\cap U_T\ne \varnothing$ for $S,T\in \{X,Y,Z\}$). We may use the shorthand $U=U_X\cup U_Y\cup U_Z\in \mathcal{U}$. For simplicity, we assume that all variables are discrete with arbitrary finite supports. When appropriate probability densities exist, extending the proposed definitions to continuous or mixed random variables is straightforward. Extending estimators of the proposed measures to the continuous case is less straightforward and is not presently considered. In general, let p be the probability mass function (pmf) for all variables in the graph (i.e., $X,Y,Z,U\sim p$), capital letters represent random variables, and lowercase letters represent their realizations. For example, $p(x\mid p{a}_X)$ gives the conditional probability of the event $X=x$ given that its parents took on values $p{a}_X$. We further assume that p satisfies the causal Markov condition with respect to $\mathcal{G}$[12], with $p(x,y,z,u)=p\left(u\right)p(x\mid u_X)p(z\mid x,u_Z)p(y\mid x,z,u_Y)$. We use a hat to indicate the $do$-operator, which represents taking the action of forcing a variable to assume a particular value by means of intervention. For example, $p(y\mid \widehat{z})=p(y\mid do(Z=z))$ gives the probability of y given that Z is forced to take the value z, irrespective of the probability with which that value occurs. When working with distributions utilizing the $do$-operator, a set of rules known as the $do$-calculus can be used to identify if and how the interventional distributions correspond to observational distributions that do not utilize the interventions. While the reader is referred to Section 3.4 in [12] for the complete $do$-calculus, we provide a description of the rule which enables swapping interventions for observations in Appendix A.

The entropy of a random variable Y and conditional entropy of Y given X are given respectively by $H\left(Y\right)=-{\sum }_{y}p\left(y\right)logp\left(y\right)$ and $H(Y\mid X)=-{\sum }_{x,y}p(x,y)logp(y\mid x)$. It is worth noting that the conditional entropy yields the expected uncertainty of Y given X, and is not to be confused with $H(Y\mid X=x)=-{\sum }_{y}p\left(y\right)logp(y\mid x)$, which gives the uncertainty of Y when conditioning on a particular value of X. For two distributions p and q over $\mathcal{Y}$, the KL-divergence (also known as relative entropy) from p to q represents the excess number of bits needed to represent Y if the distribution is assumed to be q when it is in fact p, and is given by $D(p\left(Y\right)\mid \mid q\left(Y\right))={\sum }_{y}p\left(y\right)logMMp\left(y\right)/q\left(y\right)$[27,28]. The KL-divergence is zero if and only if $p\left(y\right)=q\left(y\right)$ for all y for which $p\left(y\right)>0$, and is deemed infinite if there exists a y such that $p\left(y\right)>0$ and $q\left(y\right)=0$. We use $Bern\left(\alpha \right)$ to represent the distribution of a Bernoulli random variable with parameter $0<\alpha <1$. For the KL divergence between two Bernoulli random variables with parameters $\alpha$ and $\beta$, we will use the shorthand $D(\alpha \mid \mid \beta )$. Finally, the mutual information (MI) between X and Y is given by $I(X;Y)=H\left(Y\right)-H(Y\mid X)={\sum }_{x}D(p(Y\mid X=x)\mid \mid p\left(Y\right))p\left(x\right)$. These equivalent definitions of MI give rise to two interpretations: (i) the average reduction in uncertainty in Y obtained by conditioning on X and (ii) the average increased ability to predict Y resulting from conditioning on X. It is worth noting that (barring some technical details), these definitions can be applied to continuous valued random variables by substituting integrals for sums and probability density functions for pmfs.

Building upon the work of Robins and Greenland [29], Pearl [26] formalized definitions of direct and indirect effects in the context of graphical models. Such a distinction is useful in disentangling the mechanisms via which causal influences occur. A canonical example is presented by Hesslow [30], wherein a birth control pill is suspected of directly increasing the likelihood of thrombosis in women, while simultaneously reducing thrombosis through its prevention of pregnancy (which is positively linked to thrombosis). In each of Pearl’s definitions, the magnitude of the causal effect is specified for a specific value x and is measured with respect to a reference (or baseline) value $x^{*}$. The simplest of these measures is the total effect (TE) of $X=x$ on Y given by $\mathbb{E}[Y\mid \widehat{x}]-\mathbb{E}[Y\mid {\widehat{x}}^{*}]$. The TE yields the answer to a very concise causal question, namely "How much would we expect the value of Y to change if we were to change X from $x^{*}$ to x?" As indicated by the name, the TE does not distinguish effects that x has on Y directly from those that occur via a mediating variable Z. As such, Pearl proceeds to define the controlled direct effect (CDE) of x on Y with mediator z as $\mathbb{E}[Y\mid {\widehat{x}}^{*},\widehat{z}]-\mathbb{E}[Y\mid \widehat{x},\widehat{z}]$. Once again, this measure addresses a clear causal question: "How much would we expect the value of Y to change if we were to change X from $x^{*}$ to x, but kept Z at a fixed value z?" While this is an intuitive notion of direct effect, it is important to note that it requires the intervention $do(Z=z)$. Given that it may be of interest to know the direct effect that occurs when the mediating variable is not controlled for, Pearl defines the natural direct effect (NDE) as $\mathbb{E}[Y\mid \widehat{x},Z_{x^{*}}]-\mathbb{E}[Y\mid {\widehat{x}}^{*}]$, where $Z_{x^{*}}$ is the value Z would have taken had X been $x^{*}$. Using this notion of simultaneously assigning a value to X and allowing Z to take the value it would under a different X, Pearl defines the natural indirect effect (NIE) as $\mathbb{E}[Y\mid {\widehat{x}}^{*},Z_x]-\mathbb{E}[Y\mid {\widehat{x}}^{*}]$. In words, the natural indirect effect represents the expected change in Y resulting from changing Z from the value it would take under $x^{*}$ to the value it takes under x while leaving X fixed at $x^{*}$.

While there is a considerable body of work developing IT techniques for measuring causal influence, we here focus on information flow [3] and causal strength [2].

Consider a DAG $X\to Y$ with joint distribution over nodes $X,Y\sim p$, and for the sake of exposition, assume there are no confounding variables. In this simple scenario, when considering the effect of X on Y, we can freely exchange interventions for observations (assuming we only consider x s.t. $p\left(x\right)>0$), and thus the ACE of x with respect to baseline $x^{*}$ is given by $\mathbb{E}[Y\mid \widehat{x}]-\mathbb{E}[Y\mid {\widehat{x}}^{*}]=\mathbb{E}[Y\mid x]-\mathbb{E}[Y\mid x^{*}]$. Once again, this addresses the question of how much the value of Y is expected to change as a result of switching from $x^{*}$ to x. With regard to the CS and IF methods discussed above, both would quantify the effect of X on Y as $I(X;Y)$. Consider the SMI $I_1(x;Y)$ as a measure of the specific causal influence of x upon Y and note the following:

(I) We have the equivalence $I(X;Y)=\mathbb{E}\left[I_1(X;Y)\right]$, where the expectation is taken with respect to X. As such, we can think of the specific causal effect as a random variable, whose expectation is the mutual information. In doing so, we are able to capture that different values of X may have different magnitudes of causal effect on Y, with each of those effects occurring with some probability according to $p\left(x\right)$. Moreover, this makes clear that the perspective adopted here is consistent with that of other IT measures.

(II)$I_1(x;Y)$ is non-negative for all $x\in \mathcal{X}$. Whereas a negative ACE has the clear interpretation of x causing a decrease in the expected value of Y, we are measuring influences that x has on the distribution of Y. Given that there is no obvious notion of a (potentially negative) difference between distributions, we utilize a definition that results in all causal effects having positive magnitude. This serves as a partial justification for using $I_1$, rather than $I_2$, as a foundation.

(III) The SMI does not depend on the value of Y, standing in contrast with the information of a single event $i(x;y)\triangleq log\frac{p(y\mid x)}{p\left(y\right)}$ introduced by Fano [39] and its variants, referred to as “local information measures” [40,41]. Interpreting local information measures as measures of causal influence is challenging given that they are negative when Y takes on values that are unexpected given $X=x$. We adopt the perspective that, while different values of X may have different levels of effect on Y, they can only affect the distribution of Y, with the specific value y occurring randomly according to an appropriate conditional (or interventional) distribution.

(IV) The SMI does not require specifying a reference value $x^{*}$. Instead, we can view SMI as measuring the causal effect of x as compared with the X that would have occurred naturally. This suggests an intuition for the appearance of IT measures of causal influences in complex natural networks—values of X that are seen as changing the course of nature will be assigned a large causal influence. Given that we can (in this setting) exchange observation for intervention, we can view the SMI as comparing the effect of an intervention $\widehat{x}$ with a random (i.e., non-atomic) intervention $\widehat{X}$ with $X\sim p$ (see [12,42] for discussions on random interventions).

(V) The SMI addresses a very clear causal question: “How much different would we expect the distribution of Y to be if, instead of forcing X to take the value x, we let X take on a value naturally?” Stated more compactly: “How much would we expect performing the intervention $do(X=x)$ to change the course of nature for Y?”

(VI) We can interpret the SMI as comparing a ground truth distribution of Y conditioned on x ($p(Y\mid x)$) with a counterfactual distribution wherein nature was allowed to run its course ($p\left(Y\right)$). This works well with the interpretation of the KL-divergence as a measure of excess bits resulting from encoding Y using the distribution that is not the true distribution from which Y is sampled. The use of the KL-divergence is further justified in this context by the fact that the logarithmic loss is unique in its ability to capture the benefit of conditioning on X in the prediction of Y [43].

(VII) Finally, we note that $I_1(x;Y)=0$ if and only if $p(y\mid x)=p\left(y\right)$ for all y for which $p\left(y\right)>0$. By contrast, it is possible to have $I_2(x;Y)=0$ and $p(y\mid x)\ne p\left(y\right)$. The following example illustrates why this is undesirable:

Following the process of [26], we here formalize a series of definitions of total/direct/indirect causal influences from an information theoretic perspective. When leaving the comfort of the unconfounded two-node DAG, it is necessary to incorporate the interventions in the definition of the causal measures:

With the exception of the interventional notation, the STE is equivalent to the SMI. Note that for a DAG given by $X\to Y$, we will have $STE(x\to Y)=I_1(x;Y)$ but $STE(y\to X)=0\ne I_1(y;X)=D(p(X\mid y)\mid \mid p\left(X\right))$, where $STE(y\to X)$ represents the specific total effect of y on X. Thus, the STE answers the question posed above in point (IV): “How much would we expect performing the intervention $do(X=x)$ to change the course of nature for Y?”

Next we define the specific controlled direct effect (SCDE) of x on Y. Given that computing the controlled direct effect must be done by means of intervention on Z, we define the SCDE with respect to a specific value z, as it is unclear what distribution over Z should be used if the definition were to take an expectation over all possible values of z (see Theorem 2).

The SCDE measures how much we would expect performing the intervention $do(X=x)$ to alter the course of nature for Y given that Z is held fixed at z.

Next, the specific natural direct effect measures the direct effect of x on Y that occurs naturally when the mediator is not controlled:

It is helpful to dissect the two distributions of Y considered by the SNDE. Expanding the first argument as ${\sum }_{z^{\prime }}p(z^{\prime }\mid \widehat{x})p(Y\mid \widehat{x},z^{\prime })$, both distributions are given by a weighted combination of the distribution of Y conditioned upon different values of Z. In both cases, these values of Z are weighted by the probability with which they would occur under the intervention $\widehat{x}$. For the intervened values of X used to evaluate the probability of Y, however, the first distribution uses the “ground truth” value x, whereas the second uses the “naturally occurring” $x^{\prime }$, weighted according to $p\left(x^{\prime }\right)$. We can interpret the SNDE as a measure of how much we expect performing the intervention $do(X=x)$ to directly alter the course of nature for Y. Using the same logic, we can define a specific natural indirect effect:

Conducting a similar dissection, we see that the roles of x and $x^{\prime }$ are swapped from the SNDE—the “ground truth” x is used to evaluate the probability of Y, while the naturally occurring $x^{\prime }$ is used to weight different values $z^{\prime }$. As such, the only difference between the first and second arguments of the SNIE is how the value of the mediating Z is determined, resulting in a measurement of the indirect effect of x on Y. We can interpret the SNIE as a measure of how much we expect performing the intervention $do(X=x)$ to indirectly alter the course of nature for Y.

Unfortunately, the proposed definitions of SNDE and SNIE yield no obvious inequalities with respect to the STE (for example, $SNDE(x\to Y)+SNIE(x\to Y)\nleqq TE(x\to Y)$ in general). While this is initially unintuitive, it can be justified by the decision to have all causal influences be assigned a non-negative magnitude. As such, we would expect that contradictory indirect and direct effects could individually have a large magnitude while still resulting in a total effect of zero.

We now analyze the relationship between the proposed specific measures and IF/CS.

A proof is provided in Appendix B.1. The above theorem shows that the expected STE recovers the standard (unconditional) IF from X to Y. Notably, the expected STE is not equivalent to the CS associated with any subset of the arrows in the graph. Next, we show that both IF and CS provide a notion of expected SCDE:

A proof is provided in Appendix B.2. This theorem clarifies the point made earlier with regard to the value of a measure of natural direct effect. In particular, when taking an average with respect to possible control values for the mediator Z, it is not clear what distribution over Z should be used.

Even though the above causal measures are defined for specific values of X, they provide a notion of average causal influence in that they are implicitly averaging over all possible covariates U. Given that different values of u may significantly affect the nature of the relationship between x and Y, we define conditional versions of the above definitions for a specific value $U=u$. We here consider the general case where only a subset of the covariates $\tilde{U}\subseteq U$ are observed:

This definition violates the locality postulate (P2) of Janzing et al. [2] in that the causal effect of x on Y may be dependent upon how X is affected by its own parents. Allowing this is, however, consistent with the perspective that IT measures quantify the deviance from the course of nature in that the value u dictates the current natural state. Nevertheless, the terms $p(x^{\prime }\mid \tilde{u})$ and $p(x^{\prime }\mid u_X)$ can be replaced with $p\left(x^{\prime }\right)$ if one wishes to remain faithful to the locality postulate (though not explored presently, this would provide us with a notion of specific causal strength). The conditional versions of SCDE, SNDE, and SNIE follow very similar logic to that of the STE, and are defined in Appendix C.

When U is partially observable or unobservable, the nature of the dependence relationships between $U_X$, $U_Y$, and $U_Z$ will dictate the ability to estimate the proposed causal measures from observational data—more specifically, the ability to determine the interventional distributions given only estimated conditional distributions. This is crucially important given that performing interventions in many complex natural systems is infeasible. The following theorem uses the d-separation criterion [44,45] to identify when the conditional specific measures can be estimated in the partially observable setting where only $\tilde{U}\subset U$ can be observed:

The proof uses a direct application of Pearl’s $do$-calculus (theorem 3.4.1 in [12]), and is provided in Appendix B.3. By letting $\tilde{U}=\varnothing$, identifiability conditions for the specific unconditional causal effects are obtained. Similarly, the theorem provides the corollary that the conditional specific causal effects may be estimated from observational data when U is fully observable. It is important to note that the above theorem assumes that each conditional distribution can be sufficiently well estimated. Indeed, the "increased resolution" of the proposed measures comes at a cost in that reliable estimation of the proposed measures poses challenges for values of X that occur infrequently. Consider, for example, estimating the second argument of the KL-divergence defining the SNDE in (8), namely $p(y\mid {\widehat{x}}^{\prime },z^{\prime })$. Given that there is a sum over $x^{\prime }$ and $z^{\prime }$, it is necessary to know this distribution for every pair $(x^{\prime },z^{\prime })$. Thus, when $p(x^{\prime },z^{\prime })$ is very small, a significant amount of data will be required to estimate $p(y\mid x^{\prime },z^{\prime })$ (and therefore the SNDE) reliably.

The opacity of measuring causal influences in bits can be addressed by identifying a normalization procedure.

The normalized versions of the other specific causal measures are provided in Appendix D. For the sake of exposition, suppose $\tilde{U}=\varnothing$ and recall the data compression interpretation of $STE(x\to Y)$ as the excess number of bits used to encode Y under the assumption X occurs naturally when we have in fact forced $X=x$ by means of an intervention. Noting that $H(Y\mid do(X=x\left)\right)$ represents the number of bits required to encode Y when we have (knowingly) forced $X=x$, the denominator of $\overline{STE}$ gives the total number of bits used to encode Y under the incorrect assumption of a naturally occurring X. As such, the normalized STE represents the fraction of bits used to encode Y under the assumption that X occurred naturally that are unnecessary when performing the intervention $do(X=x)$.

As a result of the non-negativity of entropy and the KL-divergence, the normalized STE is bounded between zero and one. Interpreting $\overline{STE}$ is facilitated by considering the scenarios that yield the extremal values. First, the normalized STE is zero if and only if the STE is zero, which is to say that $p(y\mid \widehat{x},\tilde{u})=p(y\mid \tilde{u})$ for all y for which $p(y\mid \widehat{x},\tilde{u})>0$. More interestingly, the normalized STE is one if and only if the STE is greater than zero and $H(Y\mid do(X=x),\tilde{U}=\tilde{u})=0$. As such, the normalized STE being equal to one represents x having a maximal causal effect on Y in the sense that performing the intervention $do(X=x)$ determines the value of Y with 100 percent certainty. It should be emphasized that, like the unnormalized measures, this notion of maximal causal effect applies strictly in a distributional sense and says nothing of the direction or magnitude of the causal effect with respect to the units of Y. For example, if performing $do(X=x)$ results in $Y=\mathbb{E}\left[Y\right]$ with probability one, then $H(Y\mid do(X=x\left)\right)=0$ and we would conclude that x has a maximal effect on Y even though x causes Y to take the value it is expected to take absent an intervention.

For the first example consider a simple chain $X\to Z\to Y$. This can be thought of as a simplified version of the example proposed by Ay and Polani [3] and modified to include noise by Janzing et al. (example 7 in [2]). We will consider the simplest case of this example where a binary message is being passed from X to Z to Y, with the message being flipped by Z and Y with probability $ϵ$. We will interpret each variable as representing the message it passes on, i.e., $X=1$ means “X passes the message 1 to Z.” Formally, let $X,Y,Z\in \{0,1\}$ with $X\sim Bern(0.5)$: where ⊕ is the XOR operation.

Focusing first on the effect of x on Y, we note that because the only path from X to Y is the one through Z, the direct effect is zero and the total and indirect effects are equal. Noting that $Y\sim Bern(0.5)$, $Y\mid do(X=0)\sim Bern\left(2ϵ\right(1-ϵ\left)\right)$, and $Y\mid do(X=1)\sim Bern(1-2ϵ(1-ϵ\left)\right)$, the total effect is the same for both $x\in \{0,1\}$ and is given by:

Thus, as the probability of flipping the message approaches zero, Y will be deterministically linked to X, and X resolves the entire one bit of uncertainty associated with Y. Now consider the conditional STE of z on Y for a particular x. We can compute this by comparing the distributions $p(y\mid x,\widehat{z})=p(y\mid z)$ and $p(y\mid x)$. Given the symmetry of the problem, this will take one of two values depending on whether or not x and z are equal:

As $ϵ$ approaches zero, the STE approaches zero when $x=z$ and infinity when $x\ne z$. To understand this result, fix $ϵ$ to be an arbitrarily small number such that Z will pass on its received message with high probability. Thus, when $x=z$, it is, in a sense, unreasonable to endow Z with responsibility for causing the value taken by Y when it is propagating the message in a nearly deterministic manner. In such a case, it is not so much Z that is causing Y, but rather X that initiated a chain reaction. On the other hand, in the unlikely occurrence that $x\ne z$, we have that Z does have a causal effect on Y. This scenario can be thought of as Z acting of its own volition in selecting a message to pass to Y.

We acknowledge that the notion of an unbounded causal influence is initially unsettling. When looking closer, however, this property is intuitive. First, we note that for any fixed $ϵ>0$, the STE will be finite. It is only for $ϵ=0$ that the STE could be infinite, but in that case, the setting that results in infinite influence happens with probability zero. Thus, in general, an infinite influence could only be achieved through intervention. Furthermore, such an intervention would have to assign a value to a cause that occurs with probability zero, and that cause would in turn have to enable an otherwise impossible effect to have non-zero probability.

This conditional formulation violates the locality postulate (P2) of Janzing et al. [2] in that the effect of z depends on the value of its own parent, x. We do not claim that the perspective taken here is "correct," but merely point out that there exist justifications for considering the value of a cause’s parent in evaluating the causal effect.

Consider a 3-node DAG characterized by the connections $X\to Y\leftarrow Z$ with $X\sim Bern(0.5)$, $Z\sim Bern(0.1)$ and:

Given that X and Z are both parentless, we can treat interventions on X and Z as observations, and the CS, conditional IF, and conditional mutual information (CMI) are equivalent. In particular, we have that ${\mathfrak{C}}_{X\to Y}=I(X\to Y\mid \widehat{Z})=I(X;Y\mid Z)\approx 0.48$ and ${\mathfrak{C}}_{Z\to Y}=I(Z\to Y\mid \widehat{X})=I(Z;Y\mid X)\approx 0.06$. Writing CMI as a difference of conditional entropies $I(Z;Y\mid X)=H(Y\mid X)-H(Y\mid X,Z)$ provides us with the interpretation of CMI as the reduction in uncertainty of Y resulting from the added conditioning of Z, which will always be non-negative.

Next we consider $STE(x\to Y\mid z)$ and $STE(z\to Y\mid x)$ for $(x,z)\in {\{0,1\}}^2$. Given the symmetry of the problem with respect to X, we only need to consider two of the four possible values of $(X,Z)$, namely $(x_0,z_0)\triangleq (0,0)$ and $(x_0,z_1)\triangleq (0,1)$. In order to compute the STE for each X and Z to Y in either case, we need the following distributions:

For a given $(x,z)$, the STE is given by $STE(x\to Y\mid z)=D\left(p\right(Y\mid x,z)\mid \mid p(Y\mid z\left)\right)$ and $STE(z\to Y\mid x)=D\left(p\right(Y\mid x,z)\mid \mid p(Y\mid x\left)\right)$:

The results presented above are intuitive: when $z=0$, then the value taken by Y is largely determined by X, and the knowledge that $z=0$ tells us very little about the distribution of Y. On the other hand, when $z=1$, X has no bearing on the value taken by Y. Thus, in this scenario, it is the value taken by Z that has caused the shift in the distribution of Y, even though Z provides no information with regard to the particular value taken by Y. In this sense, we can think of Z as causing uncertainty in Y. This scenario makes particularly clear why it makes sense to condition on the cause but take an expectation with respect to the effect—no outcome y could be attributed to being a result of $z=1$, despite the clear influence that such an event has on the distribution of Y.

Consider a scenario where a collection of n iid variables $X_i\sim Bern\left(ϵ\right)$ collectively influence a single outcome Y, i.e., $X_i\to Y$ for $i=1,\cdots ,n$. For a given context ${\left\{x_i\right\}}_{i=1}^n$, let k be the number of $x_i$ that are one, i.e., $k={\sum }_i{x}_i$. Then let Y be distributed as: where $K={\sum }_i{X}_i$ is a random variable. One interpretation of this example is that each $X_i$ is a potential inhibitor of Y. As more inhibitors become activated (i.e., as k grows), the effect of adding another inhibitor diminishes. Since the value taken by K depends on the values taken by each $X_i$, a measure that averages with respect to $X_i$ will not capture this change in causal effect that results for different values of k.

As with the previous example, the CS, conditional IF, and CMI are equivalent for this problem setting. While there is no simple computation for these measures as a function of $ϵ$ and n, there are a couple of key points. First, the influence of each of the variables $X_i$ on Y is the same, i.e., $I(X_i;Y\mid X_1,\cdots ,X_{i-1},X_{i+1},\cdots ,X_n)=I(X_1;Y\mid X_2,\cdots ,X_n)$ for all $i=1,\cdots ,n$. Second, as $n\to \infty$, the probability of $Y=1$ goes to zero, and as $ϵ\to 0$, the probability of $Y=1$ goes to one. In either of the limits, the entropy of Y goes to zero and thus so does the causal influence of each $X_i$ as measured by either CMI, conditional IF, or CS.

Now consider a realization ${\left\{x_i\right\}}_{i=1}^n$ and the corresponding $STE(x_1\to Y\mid x_2,\cdots ,x_n)$. While the influence of each $x_i$ on Y will not be the same for a given realization, the symmetry of the problem is such that the computation will be performed in the same manner for each $x_i$. Letting $k_1\triangleq {\sum }_{i=2}^n{x}_i$ be the number of ones excluding $x_1$, define the following distributions:

Then, for a given realization, the STE is a function of $x_1$ and $k_1$:

In interpreting these results, first assume that $ϵ$ is small, meaning that for each of the inhibitors, it is unlikely that it will be activated. As a result of this assumption, we have $STE(X_1=0\to Y\mid k_1)<STE(X_1=1\to Y\mid k_1)$, i.e., an inhibitor has a greater influence when it is activated. More interestingly, note that $STE(x_1\to Y\mid k_1)$ is strictly decreasing in $k_1$. This is consistent with the intuition provided above, namely that if a large number of inhibitors are active, then they share responsibility and the influence of any single one is negligible. On the other hand, if only one is activated (i.e., $(x_1,k_1)=(1,0)$), then in the limit of $ϵ\to 0$, its influence will approach infinity (and its normalized influence will approach one).

In order to implement the proposed framework, we first need to formulate a causal DAG representation of the dataset discussed above. As a starting point, consider the DAG on the left side of Figure 4, where we let E represent an annual ENSO phase, $T_1,\cdots ,T_6$ represent the quantized two-week temperature anomaly averages for January through March (i.e., $T_1$ averages January 1st through 14th, $T_2$ averages January 15th through 28th, etc.), and U represents the other factors, such as seasonality and CO${}_2$ forcing. This DAG encodes a number of assumptions. First, it encodes the intuition that seasonality may affect ENSO and the temperature, but not the other way around. Similarly, ENSO will affect the temperature in the PNW, but not the other way around. The more interesting implicit assumption is that there is a persistence signal in the temperature represented by the arrow $T_{i-1}\to T_i$. Importantly, we have assumed that this persistence signal is Markov (when conditioned on E and U), i.e., there is no arrow $T_{i-k}\to T_i$ for $k>1$. This assumption significantly simplifies estimation of the direct and indirect effects of E on $T_i$, as those require estimating the distribution of $T_i$ for every possible combination of its parents. This serves as a motivation for the decision to consider two-week averages—if we were to simply consider daily temperatures, it is unreasonable to expect that $T_i$ would be independent of $T_{i-2}$ when conditioned on E, U, and $T_{i-1}$.

We next incorporate two assumptions in order to simplify the causal model. First, we assume that all the effects of U are removed by the detrending and removal of annual cycle performed in the preprocessing steps. It is to be expected that this assumption will hold for the well known shared causes (such as the aforementioned seasonality and CO${}_2$ forcing), but the possibility of other factors that have effects not captured by the leading six harmonics of the annual cycle is important to note. The second assumption we make is that the distribution of the temperature anomaly averages does not change over time, i.e., that $p(t_i\mid t_{i-1},e)$ and $p(t_i\mid e)$ are not dependent on i. After making these assumptions, we obtain the simplified DAG on the right of Figure 4, where we introduce the new variable S to represent the past temperature anomaly average and T to represent the subsequent temperature average, and note that this perfectly matches the mediation model in Figure 1 with $U=\varnothing$. We can think of T as representing $T_i$ and S as representing either $T_{i-1}$ or the collection $T_1,\cdots ,T_{i-1}$. To see that these interpretations of S are equivalent, consider the SNDE, given by:

Now let $T=T_i$ and $S=T_1,\cdots ,T_{i-1}\triangleq T_1^{i-1}$, and note that:

Plugging these into the second argument of the KL-divergence in Equation (15), we get:

Given that S appears nowhere in the first argument the KL-divergence, we can see that whether $S=T_{i-1}$ or $S=T_1^{i-1}$, the result is the same. The same procedure can be applied to show equivalence for the SNIE. We here choose the interpretation $S=T_{i-1}$. As a result of the assumption that $p(t_i\mid e)$ does not depend on i, we have that $p(t\mid e)=p(s\mid e)$ for $t=s$. It should be noted that for $T=T_1$ (i.e., the average for the first two weeks of January), we define $S=T_0$ to be the average taken over the last two weeks of December.

We define the dataset from which we estimate the causal influences as $\mathcal{D}={(e_n,s_n,t_n)}_{n=1}^{9840}$. Given that there is a large amount of data and a relatively small alphabet size, we utilize plug-in estimators of the proposed measures, where every distribution in question is estimated using a maximum likelihood estimator. Since E has no parents reduced DAG, we can freely exchange interventions $\widehat{e}$ for observations e in the estimation of the effect of e on T. As such, the estimates of the specific effect of ENSO on temperature are given by: where ${\widehat{p}}_{\mathcal{D}}$ gives the maximum likelihood estimate of p on the sample $\mathcal{D}$ (see Appendix E).

Next note that the conditional STE of the past temperature average S on the subsequent temperature T conditioned on an ENSO state E is:

Letting $X=S$, $Y=T$, $Z=\varnothing$, and $U=E$, it follows from Theorem 3 that we can estimate the total effect from observational data. Therefore, we use the following plug-in estimator:

Given the absence of an intuitive link between bits and temperature, we choose to focus on the normalized versions of the proposed causal measures (see Appendix E).

By applying these estimators to the complete dataset $\mathcal{D}$, we obtain point estimates of the desired measures. For ease of notation, we omit $\mathcal{D}$ from the estimates from here on. It is important to note that even though not all estimates will utilize all 9840 samples, Figure 5 makes clear there is a considerable amount of samples available for estimating every distribution in question. In particular, we see that:

In other words, the distribution estimated on the smallest number of samples is $p(t\mid E=1,S=-1)$, and this estimate is obtained from 596 samples.

In addition to these point estimates, it is desirable to have a means of measuring the significance of the estimated measures and quantifying the uncertainty in our estimates. To achieve these goals, we perform a nonparametric bootstrap hypothesis test [61] and construct a nonparametric bootstrap confidence interval [62]. The goal of the hypothesis test is to estimate the distribution of the estimated measure under a null hypothesis (H0) and assess the likelihood that our estimate came from such a distribution. In this case, H0 corresponds to the absence of a causal link, which would result in the true causal measure being equal to zero. The primary challenge to performing this test is the generation of samples from a distribution representative of H0. We accomplish this using a scheme similar to that presented in Example 2 in [2] wherein we group the data by one of the three variables (E, S, or T) and shuffle the other two in order to break one of the causal links. For example, when performing the test for the direct effect of E on T, we split the data into three sets: $\{n:s_n=-1\}$, $\{n:s_n=0\}$, and  $\{n:s_n=1\}$. Within each of these sets, we shuffle (i.e., permute) all the samples of E (or T). Because the shuffling occurs within groupings of S, any possible link from E to S and S to T is preserved (and thus so is the indirect effect), but the link between E and T is destroyed. Each of these permutations is then treated as a sample under H0 from which we estimate the SNDE. We perform this shuffling and estimation procedure 10,000 times and use the 95th percentile as the cutoff threshold for statistical significance. This threshold is given by the upper whisker on the boxplots labeled H0 in the figures in the next section. When performing this test for the indirect effect, we choose to break the link from S to T rather than from E to S in order to preserve the assumption that $p(s\mid e)=p(t\mid e)$ for $s=t$.

To quantify the uncertainty in our point estimate we construct a nonparametric bootstrap confidence interval by repeatedly drawing a collection of samples from the empirical distribution of our data and estimating the measure on the new collection of samples. Specifically, let ${\mathcal{D}}_b^{*}={(e_{j_b^n},s_{j_b^n},t_{j_b^n})}_{n=1}^{9840}$ be the bth bootstrap sample, where $j_b^n$ are drawn independently from the uniform distribution over $\{1,2,\cdots ,9840\}$ for $b=1,\cdots ,$ 10,000. We estimate the causal measure in question on each of the 10,000 bootstrap samples and use the 5th and 95th percentiles as the lower and upper bounds of the confidence interval.

We estimate the normalized STE, SNDE, and SNIE of ENSO on temperature and the normalized conditional STE of the past temperature average on the next average conditioned on ENSO. In every case, the measure is estimated on the complete dataset (red ×) and compared with the corresponding weighted average (i.e., “non-specific”) measure (red dashed lines). For the specific measure, we obtain an estimate for each value of the cause, i.e., $e\in \{-1,0,1\}$ or $s\in \{-1,0,1\}$. The average measure is then calculated by taking an expectation of the specific measures with respect to $\widehat{p}\left(e\right)$ or $\widehat{p}(s\mid e)$. As an example, the red dashed line in the left panel of Figure 6 represents ${\mathbb{E}}_{\widehat{p}\left(E\right)}\left[\widehat{\overline{STE}}(E\to T)\right]$, and the three red dashed lines in Figure 7 represent ${\mathbb{E}}_{\widehat{p}(S\mid e)}\left[\widehat{\overline{STE}}(S\to T\mid e)\right]$ for $e\in \{-1,0,1\}$. Each figure also displays two boxplots for each measure—the first shows the distribution of the measure estimated on the bootstrap samples and the second shows the distribution of the measure estimated under the null hypothesis that the causal link in question does not exist (denoted “H0”).

We begin by considering the total effect of ENSO on temperature shown in Figure 6. Given that E is a root node in the DAG representation given on the right of Figure 4, we note that STE and SMI are equivalent, i.e., $STE(e\to T)=I_1(e;T)$, and the expectation gives an estimate of the mutual information, i.e., $\widehat{I}(E;T)={\mathbb{E}}_{\widehat{p}\left(E\right)}\left[\widehat{STE}(E\to T)\right]$. This illustrates the value of considering a specific causal measure—as we can see, the estimated effect of $E=1$ is roughly three times the effect as estimated by the average with respect to E. Recall the interpretation of the SMI as a measure of how much we would expect performing $do(E=e)$ to change the course of nature for T. Under this interpretation, we see that forcing an El Niño year would alter the temperature distribution from what we would expect to occur naturally moreso than forcing a La Niña or neutral year.

Figure 6 shows that both the direct and indirect effects are less than the STE for all values e. This is consistent with the intuition that the direct and indirect effects of ENSO on temperature would not cancel each other out. Intuition is also validated by the fact that the SNIE is less than the SNDE for all values. While this need not be the case in general, we make the assumption that S and T are identically distributed given E, and thus we would expect the indirect link $E\to S\to T$ to be weaker than the direct link $E\to T$. While the proposed method does not explicitly identify a physical causal mechanism, the indirect link would represent a situation wherein certain temperatures give rise to environmental circumstances that may affect future temperatures, for example snow pack or soil moisture. Given that there is no evidence in the literature of these environmental factors having a large affect on temperatures, it is sensible that the SNIE is very low. As a final point, we note that while all estimates are statistically significant as measured by our proposed tests, only the effect of $E=1$ has a non-zero lower bound on the confidence interval for the SNIE. This serves as further justification for the measurement of specific causal influences—when simply measuring average influences with MI, CS, or IF, statistical significance testing results in an “all or nothing” test, whereas the present framework enables identifying influences that are significant for only some values of a cause.

We conclude this section with the conditional STE of past on current temperature in a specific ENSO phase, as portrayed by Figure 7. We can clearly see that there is a strong persistence in the temperature anomaly signal, i.e., that the past temperature average has a strong effect on the subsequent average, with the largest effect ($\widehat{\overline{STE}}(S=-1\to T\mid E=1)$) being roughly five times that of the effect of $E=1$. The fact that the largest effect of S on T occurs when performing the intervention $do(S=-1)$ during an El Niño year can likely be explained by the tendency for El Niño years to give rise to high temperatures. Thus, we would expect that forcing a cold spell during an El Niño would alter the course of nature moreso than, say, forcing a heat wave. Furthermore, the second largest effect is seen when $S=1$ and $E=-1$, i.e., when a heat wave is forced during a La Niña year. This result is reminiscent of the earlier example where there is a large causal influence resulting from a broken chain reaction. In this case, since we would expect an El Niño (resp. La Niña) year to assign a higher probability to a heat wave (resp. cold spell) that would then persist through the effect of S on T, intervening on S to force a cold front (resp. heat wave) will result in a large deviation from the natural behavior and thus a large causal effect. It is important to note that "forcing a cold spell" is ambiguous in that there are many different mechanisms by which one could hypothetically force a temperature. The following section includes a discussion of how these different mechanisms affect the ability to consider the estimated affect as a true causal effect or merely a measure of predictive utility. In either case, the proposed methods provide a clearer picture of how the relationship between subsequent two-week anomaly averages is modulated by ENSO phase than traditional IT methods. This suggests an area for future investigation, as two-week temperature persistence is not well studied outside of the context of persistent high pressure anomalies [63].

Any causal interpretation of the results is predicated on the assumption that there are no confounding factors not accounted for in the preprocessing steps. This assumption is less of an issue when measuring the effect of ENSO, where we only need to assume that there is no common cause for E and S or E and T (that there is no backdoor path, to be precise) beyond the seasonality, CO${}_2$ forcing, and any other phenomena captured by the leading six harmonics. When measuring the effect of past temperatures, however, this assumption is a bit more far reaching. For example, we have neglected to consider the temperatures in neighboring regions. Moreover, the explicit nature of the causal effect of S on T is more elusive than that of E on T. While it is reasonable to expect the temperature to have some causal effect in a literal sense (i.e., via the heat equation), it is likely that the estimation procedure is also capturing the effects of temperature related variables. For example, if we additionally included PNW atmospheric pressure waves in the model, we would expect these waves to be a common cause for S and T resulting in a significantly weaker (if not absent) link $S\to T$. As such, the above estimate of $\overline{STE}(s\to T\mid e)$ ought to be viewed as either a measure of predictive utility of the literal temperature, or the causal effect of a "meta variable" representative of the temperature and related quantities that are intervened upon as a whole. In any case, the present study serves as a starting point for the development of more intricate causal models relating ENSO and temperature. A potential avenue for continued work is to use the proposed framework on causal graphs learned from data rather than those prespecified using domain expertise. Development of methods for learning causal structures from data is a highly active area of ongoing research in climate science [64].

A second set of challenges arises from the need to estimate the measures for every value of the cause. While these challenges are indeed a fundamental challenge with the proposed framework, they provide an opportunity for the development of novel estimation and statistical testing techniques. On one hand, the proposed specific causal measures are necessarily more challenging to estimate than their average counterparts. On the other hand, they necessarily provide more resolution and allow for estimating separate confidence intervals for each element in the analysis. If we are trying to estimate $STE(x\to Y)$ but only have a small number of points in our dataset where $x_n=x$, then we would have very little confidence in our estimate. However, that need not discourage us from having high confidence in an estimate of $STE(x^{\prime }\to Y)$ for some $x^{\prime }$ for which we have many samples. That having been said, the proposed estimators and significance test used in the present study lack a formal analysis and leave considerable room for improvement.

As a final discussion point, we return to the comparison of information theoretic and statistical notions of causal influence. Despite having carefully formulated the proposed measures as measures of the extent to which an intervention results in a deviation from the course of nature, the results presented in this section beg the question: How useful are bits? As an absolute measure, it is worth noting that a measure in bits will be largely influenced by the number of quantization regions we select. While this can be partially addressed by the proposed normalization, there is no question that the data compression interpretation provided alongside those equations is less intuitive than a measure of, say, the number of degrees warmer we would expect it to be an El Niño year than a La Niña year. Moreover, this intuition gap would be even larger for someone outside of the information theory community (e.g., climate scientists). This is not to say that the proposed measures are so opaque that they are unusable. In fact, we believe that they provide more interpretable notions of causal influence than other information theoretic measures that have experienced some popularity in the literature. Instead, this discussion is merely intended to maximize the level of intuition that we can associate with the proposed measures while simultaneously acknowledging the limitations of information theoretic measures in terms of interpretability.