Bayesian Inference Federated Learning for Heart Rate Prediction

Abstract

The advances of sensing and computing technologies pave the way to develop novel applications and services for wearable devices. For example, wearable devices measure heart rate, which accurately reflects the intensity of physical exercise. Therefore, heart rate prediction from wearable devices benefits users with optimization of the training process. Conventionally, Cloud collects user data from wearable devices and conducts inference. However, this paradigm introduces significant privacy concerns. Federated learning is an emerging paradigm that enhances user privacy by remaining the majority of personal data on users’ devices. In this paper, we propose a statistically sound, Bayesian inference federated learning for heart rate prediction with autoregression with exogenous variable (ARX) model. The proposed privacy-preserving method achieves accurate and robust heart rate prediction. To validate our method, we conduct extensive experiments with real-world outdoor running exercise data collected from wearable devices.

A EM Algorithm for Hyperparameter Estimation for Hierarchical Bayesian Regression Model

E step: The complete data log likelihood is

$$\begin{aligned} L(\varvec{\varPhi }_0)&=\log P (\{\varvec{\beta }_i, \sigma ^2_i\}_1^n, \{\mathcal {D}_i\}_i^n |\varvec{\varPhi }_0) \\&= \log ( P(\{\mathcal {D}_i\}_i^n |\{\varvec{\beta }_i, \sigma ^2_i\}_1^n, \varvec{\varPhi }_0) P(\{\varvec{\beta }_i, \sigma ^2_i\}_1^n| \varvec{\varPhi }_0)) \\&= \log \left( \prod _{i=1}^n \text {N}(\varvec{y}_i; \varvec{X}_i\varvec{\beta }_i, \sigma ^2_i\varvec{I})\text {NIG}\left( \{\varvec{\beta }_i, \sigma ^2_i\};\varvec{\varPhi }_0\right) \right) \\&= \sum _{i=1}^n \log \left( \text {NIG}\left( \{\varvec{\beta }_i, \sigma ^2_i\};\varvec{\varPhi }_0\right) \right) + C, \end{aligned}$$

where C contains all the terms that are independent of $\varvec{\varPhi }_0$. The conditional expected complete data likelihood is:

$$\begin{aligned} Q(\varvec{\varPhi }_0|\varvec{\varPhi }_0^{t-1})&=E_{\{\varvec{\beta }_i, \sigma ^2_i\}_1^n|\varvec{\varPhi }_0^{t-1}, \{\mathcal {D}_i\}_i^n}[ L(\varvec{\varPhi }_0) ] \\&\approx \frac{1}{nL} \sum _{m=1}^{L} \sum _{i=1}^n \log (\text {NIG}(\{\varvec{\beta }_i, \sigma ^{2}_i\}^{(m)}; \varvec{\varPhi }_0) \end{aligned}$$

where $\{\varvec{\beta }_i, \sigma ^{2}_i\}^{(m)}$ denotes the m-th i.i.d. sample from $P(\varvec{\beta }_i, \sigma ^2_i|\mathcal {D}_i, \varvec{\varPhi }_0^{t-1})$, which are NIG distributed. Sampling from a NIG distribution is straightforward by a standard two step procedure by firstly sampling $\sigma ^2$ from $\text {Inv-Gamma}(a_i,b_i)$ then sampling from $\varvec{\beta }$ from $\text {N}(\varvec{m}_i, \sigma ^2\varvec{\varLambda }_i^{-1})$. Essentially, we are approximating the conditional expectation with a Monte Carlo estimator with L samples from the posterior $P(\{\varvec{\beta }_i, \sigma ^2_i\}_1^n|\{\mathcal {D}_i\}_1^n, \varvec{\varPhi }_0^{t-1})$. The EM algorithm degenerates to a Monte Carlo Expectation Maximization (MCEM) [19].

M step: the objective here is to maximize the conditional expectation, namely

$$\begin{aligned} \varvec{\hat{\varPhi }}_0&= {\mathop {\hbox {argmax}}_{\varvec{\varPhi }_0}}\, Q(\varvec{\varPhi }_0|\varvec{\varPhi }_0^{t-1})\end{aligned}$$

(7)

$$\begin{aligned}&= {\mathop {\hbox {argmax}}_{\varvec{\varPhi }_0}}\, \frac{1}{nL} \sum _{m=1}^{L} \sum _{i=1}^n \log (\text {N}(\varvec{\beta }_i^{(m)}; \varvec{m}_0, \sigma ^{2(m)}_i \varvec{\varLambda }_0^{-1} )) + \log \left( \text {G}\left( \sigma ^{-2(m)}_i; a_0, b_0\right) \right) \end{aligned}$$

(8)

where we have used the property that if $x\sim \text {Inv-Gamma}(a,b)$, then 1/x is Gamma distributed with shape and rate parameters a, b, denoted as $\text {G}(a,b)$. It is easy to see that the optimal $\hat{a}_0 ,\hat{b}_0$ w.r.t Q are just the maximum likelihood estimator of a Gamma distribution with dataset $\{\sigma ^{2(m)}_i\}_{i,m=1}^{n,L}$ (the second term of Eq. (8)). An iterative generalized Newton’s method can be used to find the ML estimator of Gamma as follows [20].

$$\begin{aligned} \frac{1}{a_0}&= \frac{1}{a_0} + \frac{\overline{\log \sigma ^{-2}} -\log (\overline{\sigma ^{-2}}) + \log a_0 - \varPsi (a_0)}{a_0^2 (1/a_0 - \varPsi '(a_0))}\end{aligned}$$

(9a)

$$\begin{aligned} b_0&= \frac{\overline{\sigma ^{-2}}}{a_0}, \end{aligned}$$

(9b)

where

$$\begin{aligned} \overline{\sigma ^{-2}} =\frac{\sum _{m=1}^{L} \sum _{i=1}^n 1/\sigma ^{2(m)}_i}{nL},\; \overline{\log \sigma ^{-2}} =\frac{\sum _{m=1}^{L} \sum _{i=1}^n \log (1/\sigma ^{2(m)}_i)}{nL}. \end{aligned}$$

Take the derivative of the Gaussian term in Eq. (8) w.r.t $\varvec{m}_0, \varvec{\varLambda }_0$ and set them to zero, we can find the estimators for $\varvec{m}_0, \varvec{\varLambda }_0$:

$$\begin{aligned} \varvec{m}_0&= \frac{\sum _{m=1}^{L} \sum _{i=1}^n \frac{1}{\sigma ^{2(m)}_i} \varvec{\beta }_i^{(m)}}{\sum _{m=1}^{L} \sum _{i=1}^n \frac{1}{\sigma ^{2(m)}_i}} \end{aligned}$$

(9c)

$$\begin{aligned} \varvec{\varLambda }_0^{-1}&= \frac{1}{nL} \sum _{m=1}^{L} \sum _{i=1}^n \frac{1}{\sigma ^{2(m)}_i}(\varvec{\beta }_i^{(m)}-\varvec{m}_0)(\varvec{\beta }_i^{(m)}-\varvec{m}_0)^{T} \end{aligned}$$

(9d)