Successively Structured Gaussian Two-terminal Source Coding

Wireless Pers Commun (2009) 48:485–510 DOI 10.1007/s11277-008-9534-x Successively Structured Gaussian Two-terminal Source Coding Hamid Behroozi · M. Reza Soleymani Published online: 20 June 2008 © Springer Science+Business Media, LLC. 2008 Abstract Multiterminal source coding refers to separate encoding and joint decoding of multiple correlated sources. Joint decoding requires all the messages to be decoded simultaneously which is exponentially more complex than a sequence of single-message decodings. Inspired by previous work on successive coding, we apply the successive Wyner-Ziv coding, which is inherently a low complexity approach of obtaining a prescribed distortion, to the two-terminal source coding scheme. First, we consider 1-helper problem where one source provides partial side information to the decoder to help the reconstruction of the main source. Our results show that the successive coding strategy is an optimal strategy in the sense of achieving the rate-distortion function. By developing connections between source encoding and data fusion steps, it is shown that the whole rate-distortion region for the 2-terminal source coding problem is achievable using the successive coding strategy. Comparing the performance of the sequential coding with the performance of the successive coding, we show that there is no sum-rate loss when the side information is not available at the encoder. This result is of special interest in some applications such as video coding where there are processing and storage constraints at the encoder. Finally, we provide an achievable rate-distortion region for the m-terminal source coding. Keywords Multiterminal source coding · m-helper problem · Partial side information · Rate-distortion region · Successive coding · Sequential coding This paper was presented in part at the IEEE International Symposium on Information Theory (ISIT), July 2006 and in part at the IEEE Vehicular Technology Conference (VTC), September 2006. H. Behroozi (B) Department of Mathematics and Statistics, Queen’s University, Kingston, ON, Canada K7L 3N6 e-mail: behroozi@mast.queensu.ca M. Reza Soleymani Electrical and Computer Engineering Department, Concordia University, Montreal, QC, Canada H3G 1M8 e-mail: msoleyma@ece.concordia.ca 123 486 H. Behroozi, M. R. Soleymani 1 Introduction The behavior of many wireless networks such as wireless sensor networks can be modeled by the Chief Executive Officer (CEO) problem [1–4]. This problem, which is closely related to multiterminal source coding problem, is an abstract model for remote sensing and distributed compression in wireless networks. In the CEO problem, a firm’s CEO is interested in an underlying source X which cannot be observed directly. The CEO deploys a team of L agents (sensors) to observe the source data sequence. The agents observe independently correlated (noisy) versions of the source and then communicate information about their observations to a single fusion center (FC) for further processing. The FC aims to form an optimal estimate of X by fusing information sequences received from the agents. The overall system is depicted in Fig. 1. The encoders are distributed and cannot cooperate to exploit the correlation between their inputs. Each encoder is subject to a transmission rate constraint. This constraint comes from the restrictions on the resources such as bandwidth and power that are available at each agent (e.g. sensor node). Thus, the key challenge in such a data gathering model is conserving the energy of the distributed nodes and maximizing their lifetime. It is shown that among the three power consumption tasks by a sensor node, i.e., sensing, data processing and communication, the most energy consumption part is related to the communication tasks. Since the sensor measurements are correlated, the correlation should be exploited to avoid redundant transmission. In fact, since the bit-rate directly impacts power consumption at a sensor node, by eliminating the data redundancy and reducing the communication load, the energy resources can be managed in an optimal manner. Therefore, an efficient source coding technique is the main requirement of a distributed network. In general, multiterminal source coding or distributed data compression refers to separate encoding and joint decoding of multiple correlated sources [5]. Over the last 30 years, significant effort has been made on finding a complete characterization of multiterminal rate-distortion region. But even concrete examples of the problem are hard to analyze. For instance, the complete characterization of the rate region for the 2-terminal source coding in the Gaussian case has been found recently [6]. Nevertheless, today multiterminal source coding is still of special interest; not only because it is an open problem of information theory [7], but also because of its application in wireless communication systems. In fact, the increasing attention given to new applications such as wireless video networks or distributed sensor networks is a reason for new interests in multiterminal source coding schemes. V1 Y1 V2 Y2 X Encoder 1 Encoder 2 Source U1 R1 U2 R2 . . . .. . . VL YL Encoder L Fusion Center (FC) X UL RL Fig. 1 The CEO model. The target data X is observed by L wireless agents (sensors) as Yi . Agents encode and transmit their observations through rate constrained noiseless channels to a FC. The FC obtains an estimate of the source X within an acceptable degree of fidelity 123 Successively Structured Gaussian Two-terminal Source Coding 487 Analog Channel Side Info Encoder Digital Channel Decoder Fig. 2 Digitally enhanced analog transmission [11] Source Coding with Side Information In the point-to-point communication there is only a single source of information at the encoder and decoder, i.e., the message at the encoder, and the received signal at the decoder. If any type of useful information is added to the encoder or decoder’s information, it would be called “side” information [8]. In many distributed applications, the receiver may have some prior information about the source. This prior knowledge which is correlated to the transmitted source is considered as the decoder side information. For instance, given a video sequence, one can predict a forthcoming frame at the decoder based on the previously decoded frames by extrapolation [9]. The predicted frame works as a side information at the decoder in order to reduce the transmission bit-rate. As an another example, assume that we want to upgrade an existing analog TV system [10,11]. Instead of replacing the whole system, a digital channel can be added to the existing system. The scenario is shown in Fig. 2. The output of the analog channel can be considered as a side information to obtain a better reconstruction of the source signal. Side information is often available to improve the rate-distortion performance of the system. Source coding with side information addresses encoding schemes that exploit the side information in order to reduce the transmission rate. 1.1 Multiterminal Source Coding The rate-distortion theory for the multiterminal source coding problem was first studied by Wyner and Ziv [12]. They obtained the rate-distortion function of a single source when the decoder can observe full resolution side information about the source. Today, Wyner-Ziv coding refers to lossy source coding with decoder side information. The important fact about the Wyner-Ziv coding is that it usually suffers rate loss compared to the lossy coding of X when the side information Y , which is correlated to the transmitted source X , is available at both the encoder and the decoder. One exception is when X and Y are jointly Gaussian with a mean squared-error (MSE) distortion measure [13]. There is no rate loss with the Wyner-Ziv coding in this case, which is of special interest in some applications such as video sensor networks since many image and video sources (after mean abstraction) can be modeled as jointly Gaussian [14]. This result is the dual of Costa’s dirty paper theorem for the channel coding with side information at the transmitter only [15]. In this case the side information is knowledge about the state of the channel, available at the encoder, but not the decoder. Costa’s dirty paper analogy could be stated as follow [15]: Consider a sheet of paper covered with independent dirt spots having normally distributed intensity. A writer wants to send a message with a limited amount of ink. He knows the location and intensity of dirt spots and can take advantage of that side information. The dirty paper, with the writer’s message on it, is then sent to the reader. It picks up more normally distributed dirt along its way to the reader. Therefore, the reader cannot distinguish the dirt spots from the ink marks applied by the writer. In [15] Costa shows that the reader can gain the same amount of information as 123 488 H. Behroozi, M. R. Soleymani X Y Decoder X Event Fig. 3 1-helper scenario in a video sensor network. Two sensors are deployed to measure an event. At the decoder, we are only interested in the signal reconstruction of the sensor X which is nearer to the event than other sensor, Y if the original dirt spots were known by the reader, too. In this work, we also focus on the Gaussian sources. The generalized Wyner-Ziv source coding for noisy encoder observations has appeared in [2]. This problem is also known as the remote source coding with decoder side information [5,16]. Similar rate-distortion analysis is presented in [3], and recently in [8]. Systematic lossy source/channel coding with uncoded side information at the decoder is presented in [10]. Oohama [17] gives the solution of the Wyner-Ziv problem with coded side information for the case of two sources. This problem is also known as the 1-helper problem. 1.1.1 One-helper Coding Scheme Consider the following scenario for the event detection: a video sensor is used to monitor an event and transmit limited rate information to a FC for further processing. Also, suppose another sensor is able to monitor the same scene from a different viewing position which is farther from the event than the first sensor and hence it does not worth to reproduce its signal at the FC. The scenario is shown in Fig. 3. Since both sensors monitor the same scene, their information is correlated. Therefore, the second sensor can provide partial side information to reduce the rate required at the first sensor and hence save transmission energy for a given distortion. In fact, the information of the second sensor can be used at the FC to obtain higher reliability and lower probability of detection error. We refer to the sensor providing side information as the helper while the sensor whose measurement is going to be reproduced is referred to as the main source [18]. This problem, which is also called the 1-helper problem, for the special case of correlated memoryless Gaussian sources and squared distortion measures is investigated by Oohama [17]. In the m-helper problem only the main sources is reconstructed while other m sources work as helpers [18–21]. 1.1.2 Two-Terminal Source Coding Scheme In this scheme, two correlated sources X and Y are separately encoded and sent to a single joint decoder, where the decoder reconstruct both sources. The corresponding coding scheme is shown in Fig. 4. X is encoded at rate R0 and Y is encoded at rate R1 . The joint decoder aims to obtain estimates of X and Y with average distortions D0 and D1 , respectively. In the previous video sensor network example, if FC aims to reconstruct both sensors’ observations, the scenario can be modeled by the 2-terminal source coding problem. Berger and Yeung [22] solve the rate-distortion problem for the situation in which the reconstruction of X must be perfect, while that of Y is subject to a distortion criterion. Oohama derives an outer region for the rate-distortion region of the 2-terminal source coding problem and demonstrates that the inner region obtained by Berger [5] and Tung [23] is partially tight [17]. Wagner et al. 123 Successively Structured Gaussian Two-terminal Source Coding n n X X ϕ0 Y ϕ0 (X ) n n Y 489 ϕ1 ϕ1 (Y ) ψ = (ψ0 ,ψ1) X Y n n Fig. 4 2-terminal source coding scheme. X and Y are two correlated sources with correlation coefficient ρ. Both sources are to be reconstructed at the decoder [6] complete the characterization of the rate region for the 2-terminal source coding problem by showing that the inner bound of Berger and Tung is also tight in the sum-rate. Oohama extends his results to more than two sources for a certain class of m + 1 correlated sources, where m source signals are independent noisy versions of the main source, i.e., X i = X 0 + Ni , i ∈ {1, 2, . . . , m}. In other words, sources X 1 to X m are conditionally independent given the source X 0 [24]. In this work, we consider multiterminal source coding for the case of two terminals. We use the successive coding strategy to obtain an achievable rate-distortion region for the 2-terminal source coding problem. 1.2 Successive Coding Strategy Previous research on coding of multiterminal schemes are based on joint decoding of all messages. Joint decoding requires all the messages to be decoded simultaneously which is exponentially more complex than a sequence of single-message decodings [8]. The decoding process cannot begin before all messages have been received at the joint decoder. Inspired by previous work on successive coding strategy, we apply the successive Wyner-Ziv coding to the Gaussian multiterminal source coding schemes. As it is described in [8], the encoder and decoder perform as follows: Encoder When an encoder encodes a message, it considers two things. First, its observations and second, its statistical knowledge about the messages that the decoder has already received from other nodes in the network. Therefore, previously decoded messages that are available at the decoder act as the decoder’s side information. Decoder At the decoder, instead of joint decoding, messages from sources are decoded sequentially in order to increase the fidelity of estimation at each decoding step. In other words, by obtaining more data at each stage, the decoder has additional side information to use in the next decoding step. This is a decentralized strategy because at each stage, only the knowledge sharing between each encoder and decoder is needed [8]. Its successive structuring provides flexibility to deal with distributed signal processing. By applying the successive coding, the available practical Wyner-Ziv coding techniques are applicable to more general distributed and multiterminal source coding problems. Hence, this strategy simplifies the analysis of the multiterminal source coding problem by decomposing the problem into some successive Wyner-Ziv coding cases. It allows us to derive an achievable rate region for multiterminal schemes. The successive coding approach is well-known in the research community. Some of the first applications of the successive coding are as follows: 123 490 H. Behroozi, M. R. Soleymani X Encoder 0 Dec 0 X Encoder 1 Dec 1 X Fig. 5 Successive refinement coding 1.2.1 Successive Refinement Source Coding In the successive refinement source coding [25–27] at first, the source will be described by a few bits of information. The description of the source can be improved when more bits of information are available. The two-stage successive refinement scenario is depicted in Fig. 5. Source X is encoded and transmitted through a rate-constraint channel. With rate R0 , decoder 0 reproduces X with distortion level D0 . At the next stage, encoder 1 sends a sequence at rate R1 to decoder 1. With both received sequences, decoder 1 will produce a more accurate reconstruction of X at distortion level D1 . The source is called successively refinable if the two rates simultaneously lie on the rate-distortion curve, i.e., R0 = R X (D0 ), R0 + R1 = R X (D1 ) (1) where R X (D) is the rate-distortion function of the source X at distortion level D. In general, a source is successively refinable if encoding in multiple stages incurs no rate loss as compared with optimal rate-distortion encoding. It is shown that Gaussian sources with the squared error distortion measure are successively refinable [26]. Steinberg and Merhav [28] consider the problem of successive refinement in the presence of side information. They answer the question whether such a progressive encoding causes rate loss as compared with a single stage Wyner- Ziv coding. It is shown that the jointly Gaussian sources (with the squared error distortion measure) are successively refinable in the Wyner-Ziv setting. It is also shown that there is no rate loss when the difference between the source and the side information is Gaussian and independent of the side information [29]. The characterization of the rate-distortion region for successive refinability for more than two-stage systems is presented in [30]. 1.2.2 CEO Problem The objective of the CEO problem, briefly presented in Sect. 1, is to determine the minimum achievable distortion under a sum-rate constraint. By sum-rate, we mean the total rate at which the agents may communicate information about their observations to the CEO. The special case of Gaussian source and noise statistics with a MSE distortion is called the quadratic Gaussian CEO problem [31]. For this case, the rate region is known [19,20,32,33]. Applications of the successive coding strategy in the CEO problem are presented in [8,34,35]. It is shown that the successive coding is an optimal coding strategy in the Gaussian CEO problem in the sense of achieving the sum-rate distortion function [36]. Since the application of successive coding strategy into the 2-terminal source coding problem is similar to the sequential coding of correlated sources [37], this coding scheme is described as follows. 123 Successively Structured Gaussian Two-terminal Source Coding 491 X Encoder 0 Dec 0 X Y Encoder 1 Dec 1 Y Fig. 6 Sequential coding of correlated sources 1.2.3 Sequential Coding of Correlated Sources Assume that we have two correlated sources X and Y . In the sequential coding scheme, as described in [37], the first encoder observes only the source X and encodes it at rate R0 such that the first decoder is able to reconstruct the source X with distortion D0 . Then, the second encoder, who has access to the source X , observes source Y and encodes it (with the encoding allowed to depend on X ) at rate R1 so that the second decoder, with the available side information from the first decoder, can reconstruct the source Y with distortion D1 . The coding scheme is depicted in Fig. 6. When the sources X and Y are the same source, the above coding scheme reduces to the successive refinement source coding scheme [37]. In this work, we evaluate the performance of the successive coding strategy for the problem of multiterminal lossy coding of correlated Gaussian sources. We address the problem from an information theoretic perspective and determine the rate region for 1-helper source coding scheme and 2-terminal source coding problem. Our main contributions can be summarized as follows: we show that for both the Gaussian 1-helper problem and the Gaussian 2-terminal source coding problem, successively structured Wyner-Ziv codes can achieve the rate-distortion bounds. Therefore, the high complexity optimal source code can be decomposed into a sequence of low complexity Wyner-Ziv codes. We support our derivatives by geometric sphere-packing arguments [38,39]. By comparing the results of the successive coding strategy for the 2-terminal source coding and the sequential coding of correlated Gaussian sources, we demonstrate that there is no sum-rate loss when the output of the first encoder is not available at the second encoder. The result is of special interest in some applications such as video coding where there are storage constraints at the encoder. This successive coding approach leads us to derive an inner bound for the rate region of the m-terminal source coding where all m correlated sources are reconstructed at the FC with specified distortions. The rest of this paper is organized as follows: Section 2 presents the system model and problem formulation. In Sect. 3, we use the successive coding strategy in order to obtain the rate-distortion functions of both 1-helper and 2-terminal source coding systems. The achievable rate-distortion region of the successively structured m-terminal source coding system is derived in Sect. 4. Section 5 concludes this paper. 2 Problem Formulation Let X and Y be correlated Gaussian random variables such that {(X t , Yt )}∞ t=0 are jointly stationary Gaussian memoryless sources. For each observation time t = 1, 2, 3, . . . , the random pair (X t , Yt ) takes a value in real space X × Y and has a probability density function (pdf) p X,Y (x, y) of N ∼ (0, ) where the covariance matrix  is given by 123 492 H. Behroozi, M. R. Soleymani =
σ X2 ρσ X σY ρσ X σY σY2  , −1 < ρ < 1. (2) ∞ We represent n independent instances of {X t }∞ t=1 and {Yt }t=1 by data sequences n n X = {X 1 , X 2 , . . . , X n } and Y = {Y1 , Y2 , . . . , Yn }, respectively. The 2-terminal coding system is shown in Fig. 4. The correlated sources are not co-located and cannot cooperate to directly exploit their correlation. Data sequences X n and Y n are separately encoded to ϕ0 (X n ) and ϕ1 (Y n ). The encoder functions are defined by ϕ0 : X n → C0 = {1, 2, . . . , C0 } , ϕ1 : Y n → C1 = {1, 2, . . . , C1 } . (3) The coded sequences are sent to the FC with the rate constraints 1 log Ci ≤ Ri + δ, i = 0, 1 n (4) where δ is an arbitrary prescribed positive number. The decoder observes (ϕ0 (X n ) , ϕ1 (Y n )), decodes all messages, and makes estimates of all sources. The decoder function is given by ψ = (ψ0 , ψ1 ) where its components are defined by ψ0 : C0 × C1 → X n , ψ1 : C0 × C1 → Y n . (5) Let d0 : X 2 → [0, ∞) and d1 : Y 2 → [0, ∞) be the squared distortion measures, i.e., ) = (Y − Y )2 . For the reconstructed signals  d0 (X,  X ) = (X −  X )2 and d1 (Y, Y Xn = n n n n n  = ψ1 (ϕ0 (X ) , ϕ1 (Y )), the average distortions 0 , 1 can ψ0 (ϕ0 (X ) , ϕ1 (Y )) and Y be defined by  n  1 0 = E d0 (X t ,  Xt ) , n t=1  n  1  1 = E d1 (Yt , Yt ) . n t=1 For given distortion levels D0 and D1 , a rate pair (R0 , R1 ) is admissible if for any δ > 0 and any n ≥ n 0 (δ) there exists a triple (ϕ0 , ϕ1 , ψ) satisfying (3)–(5) such that i ≤ Di + δ for i = 0, 1. The rate-distortion region R(D0 , D1 ) can be defined as R(D0 , D1 ) = {(R0 , R1 ) : (R0 , R1 ) is admissible} . (6) 2.1 One-helper Problem Formulation Assume that distortion level D1 is sufficiently large, i.e., the goal is only to reconstruct X , and the other source, Y , is used as a helper. In other words, the helper will not be reconstructed, and it is just used as auxiliary information to reconstruct the main source. This problem is called the 1-helper problem. The associated coding system is depicted in Fig. 7. The encoder functions (ϕ0 , ϕ1 ) and the decoder function ψ0 for the 1-helper scenario are as follows: 123 ϕ0 : X n → C0 = {1, 2, . . . , C0 } , ϕ1 : Y n → C1 = {1, 2, . . . , C1 } , (7) ψ0 : C0 × C1 → X n , (8) Successively Structured Gaussian Two-terminal Source Coding n n X X ϕ0 Y ϕ0 (X ) n n Y 493 ϕ1 ϕ1 (Y ) ψ0 X n Fig. 7 1-helper coding system where 1 log Ci ≤ Ri + ǫ, i = 0, 1 n (9) and ǫ is an arbitrary prescribed positive number. Here, the decoder makes an estimate of the main source X n as  X n rather than estimates of both sources. There is no distortion constraint for the helper and any available rate may be used for coding of the helper information. The distortion measure for the main source X is defined as d0 : X 2 → [0,∞) which is a MSE distortion measure. The average distortion can be defined by 0 = E n1 nt=1 d0 (X t ,  Xt ) where the reconstructed signal can be represented by  X n = ψ0 (ϕ0 (X n ) , ϕ1 (Y n )). A ratedistortion pair (R0 , D0 ) is admissible if for any ǫ > 0 and any n ≥ n 0 (ǫ) there exists a pair (ϕ0 , ϕ1 , ψ0 ) satisfying (7)–(9) such that 0 ≤ D0 + ǫ. The rate region does not depend on the distortion level D1 , and it can be represented by R0 (D0 ), i.e., R0 (D0 ) = {(R0 , R1 ) : (R0 , R1 ) ∈ R(D0 , D1 ) for some D1 > 0}. It is clear that R0 (D0 ) is an outer region of R(D0 , D1 ). In [6,17] it is shown that the whole rate-distortion region of the Gaussian 2-terminal coding system can be characterized by R(D0 , D1 ) = R∗0 (D0 ) ∩ R∗1 (D1 ) ∩ R01 (D0 , D1 ), (10) where   2 1 + σX 2 2 −2R1 1−ρ +ρ 2 = (R0 , R1 ) : R0 ≥ log , 2 D0   2 1 2 2 −2R0 + σY ∗ 1−ρ +ρ 2 R1 (D1 ) = (R0 , R1 ) : R1 ≥ log , 2 D1   2 2   1 + σ X σY βmax 2 R01 (D0 , D1 ) = (R0 , R1 ) : R0 + R1 ≥ log 1−ρ , 2 D0 D1 2 R∗0 (D0 ) βmax = 1 +  1+ D0 D1 4ρ 2 × 2 2 , (1 − ρ 2 )2 σ X σY (11) (12) (13) (14) and log+ x = max{log x, 0}. Our goal is to show that the whole rate-distortion region of the 2-terminal coding system can be achieved using the successive coding strategy. There are two steps to reach this 123 494 H. Behroozi, M. R. Soleymani goal: (i) Obtaining two curved parts of the R(D0 , D1 ), i.e., (11) and (12); (ii) Obtaining the straight-line segment of R(D0 , D1 ), i.e., the sum-rate limit of (13). Note that by applying the successive coding to a multiterminal problem, the encoding and successive decoding functions can be visualized as a succession of some Wyner-Ziv stages [12]. Thus, the encoding and decoding functions for any stage of the successive coding can be described as follows. Assume that at stage l (l > 1) , Yl is the encoder observation, and Z l−1 is the side information which is received from the decoder at stage l − 1. The encoding and decoding functions are as follows: ϕl : Yln → Cl = {1, 2, . . . , Cl } , (15) n ψl : Zl−1 × Cl → Yln . (16) The coded sequences are sent to the decoder with the rate Rl satisfying 1 log2 Cl ≤ Rl + γ , n (17) where γ is an arbitrary prescribed positive number. The decoder output will be given to the next decoder at stage l + 1 as the side information. 3 Two-terminal Source Coding 3.1 Two-curved Portions of the Rate-distortion Region The two-curved portions of the rate region for the 2-terminal coding system are in fact the rate-distortion regions of the 1-helper scenarios, where one source provides partial side information to the decoder to help reconstruction of the other source signal. We determine the rate-distortion performance of the successive coding strategy in the 1-helper problem. By applying the successive coding/decoding strategy in the 1-helper problem, the problem can be decomposed into two successive coding stages. The scenario is presented in Fig. 8. Each source encodes its message while previously decoded message that is available at the decoder acts as the decoder side information. At the FC, instead of joint decoding, messages from encoders are decoded sequentially. Fusion Center n X X ϕ0 Dec 0 Z1 Dec 1 n Y Y ϕ1 Fig. 8 1-helper coding system with the successive coding strategy 123 X n Successively Structured Gaussian Two-terminal Source Coding 495 Lemma 1 Successively structured Wyner-Ziv codes can achieve the rate-distortion function of the 1-helper coding system. In other words, for every D0 > 0, the achievable rate region of the successive coding strategy for the 1-helper scenario can be represented by   2 1 + σX 2 2 −2R1 R0 (D0 ) = (R0 , R1 ) : R0 ≥ log 1−ρ +ρ 2 . (18) 2 D0 Proof Consider the encoder of source Y . The encoder observation is the source signal and there is no side information available at decoder 1. Although we intend to encode the source Y , the decoder aims to obtain an estimate of the source X . We denote the estimate of the source X at decoder 1 as  X 1 . Based on the results of [8] for Wyner-Ziv coding of noisy observations (Page 38 of [8] Eq. 3.5), the minimum required rate for transmission an of Y in order to achieve average distortion D X 1 in estimating X at the decoder 1, i.e., E{ n1 nk=1 (X k −  X 1,k )2 } ≤ D X 1 would be as follows:   2 σ X − σ X2 |Y 1 (19) R1 (D X 1 ) = log , σ X2 |Y ≤ D X 1 ≤ σ X2 2 D X 1 − σ X2 |Y   where σ X2 |Y = σ X2 1 − ρ 2 . In the second step, we consider the encoder of source X . Let Z 1 denote the output signal generated by the helper at the decoder. Then, X can be encoded at the Wyner-Ziv rate  2  σ X |Z 1 1 R0 (D0 ) = log . (20) 2 D0 To obtain σ X2 |Z 1 , we use an innovation form, given in [40], to rewrite the relationship between X 1 = (X + V )α the estimate of the source X at decoder 1, i.e.,  X 1 , and the source X as  σ2 D X1 X where V ∼ N (0, σ 2 −D ) and α = (1 − X1 X DX 1 ). σ X2 In fact, this relationship is presented in [40, p. 370] as the forward channel realization of the Gaussian rate-distortion function. This channel realization that achieves the rate-distortion function is also called the optimum test channel. This relationship gives us the distortion of E (X −  X 1 )2 = D X 1 . The message Z 1  can be considered as Xα1 = X + V , i.e., Z 1 can be considered as the source X in an additive white Gaussian noise (AWGN), V . Thus, σ X2 |Z 1 = D X 1 . By substituting this result in (20) we obtain   DX1 1 R0 (D0 ) = log , 0 ≤ D0 ≤ D X 1 . (21) 2 D0 Rewriting Eq. 19 as D X 1 = σ X2 1 − ρ 2 + ρ 2 2−2R1 , (22) and then substituting (22) in (21) results in (18). This is the achievable rate-distortion function by the successive coding strategy. Comparing our result with the results of [17] and [6] in (11) shows that by applying the successive coding strategy, the rate-distortion function for the 1-helper coding system is achievable. This completes the proof. ⊔ ⊓   2 −2R 2 2 −2R 0 1 By rewriting (18) as D0 = σ X 2 1−ρ +ρ 2 it can be seen that each additional bit of the main source description reduces the average final distortion by a factor of 4; however, each additional bit of the helper description can reduce the average distortion by at most a factor of 4. This maximum reduction occurs when the correlation coefficient is one. Also, under a constant rate of the helper, D0 is a decreasing function of the correlation coefficient. 123 496 H. Behroozi, M. R. Soleymani Remark 1 We can also obtain the result of Lemma 1 when decoder 1 decodes the associated source signal Y instead of X . Then, this decoded signal is used as the side information for decoder 0. This could be shown as follows. The encoder of Y encodes and transmits its signal with the rate   σY2 1 R1 (D1 ) = log (23) , 0 ≤ D1 ≤ σY2 2 D1 where D1 is the average distortion in estimating Y at decoder 1. The encoder of X can encode the source X by considering its statistical knowledge about the decoder’s data, i.e., Z 1 = Y , with the rate ⎛ ⎞  2  σ X2 1 − ρ X2 Y σ X |Y 1 1 ⎠, (24) R0 (D0 ) = log = log ⎝ 2 D0 2 D0  D1 σY2 where Z 1 = Y = Y + V with V ∼ N (0, σ 2 −D ). Therefore, ρ X Y = ρ σY2 − D1 /σY . By Y 1 doing some manipulations, we will see that the rate-distortion tradeoff in (24) will be the same as (18). 3.1.1 Geometric Sphere-Packing argument In this subsection, we show that the rate distortion function of the successively structured 1-helper problem can be derived from geometric sphere-packing arguments [38,39]. Barron et al. [38] use these arguments to develop a geometric interpretation of the duality between the information-embedding problem and the Wyner-Ziv problem. In particular, the authors in [38] show that the rate-distortion function of Wyner-Ziv source coding and the channel capacity in the quadratic-Gaussian case can be derived geometrically. Similar arguments are also developed in [39] to illustrate the duality between channel coding and rate-distortion with side information. These arguments for the Wyner-Ziv source coding with noisy observation are presented in [8]. The basic geometric interpretations, presented in [8,38,39], for deriving the rate-distortion function of the Wyner-Ziv scheme can be described as follows. Given a side information Y at the decoder, a minimum mean-squared error (MMSE) estimate of the source is E[X |Y ] and the associated estimation error is σ X2 |Y . Before exploiting the message from the encoder, thedecoder knows that the source vector X lies within an uncertainty sphere of radius ru = nσ X2 |Y centered at its source estimate based on the side information Y only, i.e., E[X |Y ]. This sphere is depicted by a large dottedcircle in Fig. 9. Within this sphere, we pack spherical quantization regions of radius rq = n D − σ X2 |Y . The encoder maps X to the label of the quantization region in which it falls. The decoder picks out the quantization vector from the uncertainty sphere by referring to the label of the quantization region. We apply these geometric interpretations into the successively structured 1-helper problem. Lemma 2 The rate-distortion function of the successively structured 1-helper problem can be derived using geometric sphere-packing arguments. Proof Consider the successively structured 1-helper problem depicted in Fig. 8. At the first step, we obtain the transmission rate of the side information Y . Before receiving any information from the encoder, the decoder 1 only knows the transmitted vector lies in an uncertainty 123 Successively Structured Gaussian Two-terminal Source Coding 497 ru X rq Fig. 9 Sphere-packing argument to determine the rate-distortion function. The source estimate lies within the dotted circle with the radius of ru . The solid small circles with the radius rq correspond to quantization regions at the encoder    sphere of radius ru 1 = n σ X2 + ε11 which is centered at the source estimate E [X ]. The encoder of the helper source maps Y to the label of the quantization region in which it lies. Since our final goal is to reconstruct X at the decoder, we must consider how much the encoded message once decoded will help to reconstruct X . Therefore, the radius of each spherical  quantization region is equal to rq1 = σ2 n σ 2 −σX 2 X X |Y D X 1 − σ X2 |Y − ε22 . If no two quantiza- tion regions share the same label in each large sphere of radius ru 1 , the quantization region in which Y is located can be determined without error. The scenario is depicted in Fig. 9. We determine the minimum number of quantization spheres required to cover the uncertainty   n sphere. This number is lower bounded by the ratio of volumes of spheres: M1 ≥ k(n) ru 1  n k(n) rq1 where k(n) is a coefficient that is a function of the dimension. Thus, the lower bound on the rate-distortion function R1 (D X 1 ) can be obtained as ⎞ ⎛  2  2 σ X − σ X2 |Y σ X + ε11 1 1 ⎟ 1 ⎜ R1 = log2 M1 ≥ log ⎝ . ⎠ > log σ X2 n 2 2 D X 1 − σ X2 |Y 2 −ε D − σ X1 22 X |Y σ 2 −σ 2 X X |Y (25) In the next step, based solely on the available side information at the decoder 0, Z 1 , the decoder can have a MMSE equal to σ X2 |Z 1 = D X 1 . Therefore, the transmitted vector X falls 123 498 in an uncertainty sphere of radius ru = H. Behroozi, M. R. Soleymani  √ n σ X2 |Z 1 + ε1 = n (D X 1 + ε1 ) which is cen- tered at the source estimate E [X |Z 1 ]. This sphere is shown by the dotted circle in Fig. 9. The encoder 0 maps X to the label of the quantization region in which it lies, where the radius √ of each quantization sphere equals to rq = n (D0 − ε2 ). Thus, the minimum number of quantization spheres of radius rq required to cover the larger sphere of uncertainty is lower bounded by √ n k(n) n (D X 1 + ε1 ) M≥ (26) √ n . k(n) n (D0 − ε2 ) As a result, the lower bound on the rate-distortion function of the 1-helper problem can be derived as     D X 1 + ε1 DX1 1 1 1 R0 = log2 M ≥ log > log . (27) n 2 D0 − ε2 2 D0 By substituting D X 1 from (25) into (27), the same result as (18) will be obtained. ⊔ ⊓ To apply the successive coding strategy into the 2-terminal coding system, there are two possible orderings for the coding/decoding process: (a) the message from source Y is decoded first, and (b) the message from source X is decoded first. Ordering (a) determines the ratedistortion function for the 1-helper problem when Y is the helper and therefore the distortion for reconstructing X is minimized. Ordering (b) determines the rate-distortion function for the 1-helper problem when X is the helper and therefore the distortion for reconstructing Y is minimized. These two rate-distortion functions determine two curved portions of the rate region R(D0 , D1 ) for the 2-terminal source coding problem. 3.2 Sum-rate Limit for the 2-terminal Rate Region Now we want to obtain the last part of the R(D0 , D1 ) which is defined by the boundary of the sum-rate. Using (23) and (24), ordering (a) gives the minimum sum-rate of   2 ρ2  σ X2 σY2  σ 1 R0 + R1 = log 1 − ρ2 + X . (28) 2 D0 D1 D0 By symmetry, ordering (b) gives the minimum sum-rate of    σY2 ρ 2 σ X2 σY2  1 2 1−ρ + R0 + R1 = log . 2 D0 D1 D1 (29) By allowing time-sharing between ordering (a) and (b), the achievable sum-rate can be expressed as     α   σ X2 ρ 2 σ X2 1 − ρ 2 + D0 ρ 2 σY2 σ X2 σY2  1 1 2   1−ρ + , (30) + log R0 + R1 = log 2 D0 D1 D0 2 σY2 1 − ρ 2 + D1 ρ 2 σ X2 where 0 ≤ α ≤ 1. Since the sum-rate of (30) is achievable, it is an upper bound for the sum-rate part in the rate region of 2-terminal source coding problem. By subtracting (30) from (13) and doing some manipulations, we can show that the sum-rate distortion function of the 2-terminal source coding is achievable if 123 Successively Structured Gaussian Two-terminal Source Coding max ρ2 = min   D0 D1 , σ X2 σY2 D0 D1 , σ X2 σY2   499 −1 (31) . −1 In this case, the successive coding strategy degrades to the no-helper problem where only the source with the minimum average distortion Di (i = 0, 1) should be encoded. In our analysis, so far, we have assumed that each decoder desires to reconstruct the corresponding source within the given fidelity, Di . But we can develop connections between the data fusion and the source coding with side information. In this scenario, the encoder quantizes and encodes the source to a degree of fidelity less than Di to reduce the transmission rate. But this reduction does not affect the quality of the reconstructed signals at the FC. In fact, the remaining correlation among the decoded signals enables the FC to reconstruct the sources into desired degrees of fidelity. By using a linear estimator after the source decoder, both source signals can be reproduced at the desired degrees of fidelity, Di ’s. The joint design of the source encoding and the data fusion will yield substantial performance gain over decoupled designs. Based on this scenario, we demonstrate that the successive coding strategy can achieve all the rate-distortion tuples (R0 , R1 , D0 , D1 ) belonging to the inner bound of Berger and Tung [5,23]. In fact, the whole rate-distortion region of the 2-terminal source coding problem can be characterized by applying the successive coding strategy. Theorem 1 Successive coding strategy can achieve the sum-rate distortion function of the 2-terminal source coding problem. Proof Our proof is similar to the proof of source-splitting method presented in [41,42]. Consider the ordering (a) for the 2-terminal source coding, depicted in Fig. 10. The source encoder ϕ1 , quantizes Y n and then compresses the quantized signal. The output message W1 is transmitted at rate R1 . At the decoder side, W1 is decompressed and used to reconstruct Y n as Y n . By exploring the remained correlation between Y n and X n , the encoder ϕ0 compresses the quantized version of X n at rate R0 . Using Y n as the side information, the decoder ψ0 decodes the received signal to X n . Given X n and Y n , the linear estimator reproduces X n and Y n using linear combination of the inputs:  X n = α0 Y n + β0 X n ,  Y n = α1 Y n + β1 X n . X X n ϕ0 n Y Y ϕ1 R0 R1 ψ0 n ~ X Linear ~n ψ1 (32) Y X Estimator (FC) Y n n Fig. 10 Block diagram of the successive coding strategy (with decoding ordering (a)) in the 2-terminal source coding problem. The linear estimator fuses both received signals to produce the estimates of both sources 123 500 H. Behroozi, M. R. Soleymani Define the quantization errors as E 0n = X n − X n and E 1n = Y n − Y n . We first derive the estimator coefficients. The average quantization distortions 0 , 1 can be defined by  n   n   1 1 2 2 (X t −  Xt ) = E E 0,t , 0 = E n n t=1 t=1  n   n  1 1 2 2 (Yt − Yt ) = E E 1,t . 1 = E n n t=1 t=1 Due to the orthogonal properties for optimal estimations, applying the projection theorem results in:   E (X −  X ) X = 0, E (X −  X )Y = 0,    )Y = 0. E (Y − Y ) X = 0, E (Y − Y (33) As a result, bydoing some manipulation, the estimator coefficients that minimize the average  )2 can be obtained as distortions E (X −  X )2 and E (Y − Y   2 σY + 1 σ X2 − ρ 2 σ X2 σY2 ρσ X σY 0 α0 = , β0 = ∗ ∗  2  2 2 2 2 σ + 0 σY − ρ σ X σY ρσ X σY 1 α1 = X , β1 = , (34) ∗ ∗    where ∗ = σ X2 + 0 σY2 + 1 − ρ 2 σ X2 σY2 . Based on the rate-distortion theory and Wyner-Ziv results, the transmission rates of this scheme can be expressed as n R0 ≥ I (X n ; X n ) − I ( X n ; Y n ), n R1 ≥ I (Y n ; Y n ). (35) (36)   2 For jointly Gaussian random variables X and Y , I (X ; Y ) = − 21 log 1 − ρ . We just need to obtain the correlation coefficients ρ X X , ρ X Y , and ρY Y . We can show ρ X2 X = σ X2 σ X2 + 0 , ρY2 Y = σY2 ρ 2 σ X2 σY2 2   . , ρ = XY σY2 + 1 σ X2 + 0 σY2 + 1 Thus, (35) and (36) can be computed as     σ X2 + 0 σY2 + 1 − ρ 2 σ X2 σY2 1   , R0 ≥ log 2 0 σY2 + 1   σY2 + 1 1 R1 ≥ log . 2 1 (37) (38) (39) The overall average distortions can be expressed as   X )2 = E (X − α0 Y − β0 X )2 D0∗ = E (X −      = σ X2 + α02 σY2 + 1 + β02 σ X2 + 0 − 2α0 ρσ X σY − 2β0 σ X2 + 2α0 β0 ρσ X σY , (40)   2 ∗ 2  D1 = E (Y − Y ) = E (Y − α1 Y − β1 X )     = σY2 + α12 σY2 + 1 + β12 σ X2 + 0 − 2α1 σY2 − 2β1 ρσ X σY + 2α1 β1 ρσ X σY . (41) 123 Successively Structured Gaussian Two-terminal Source Coding By substituting (34) in (40) and (41), the average distortions can be simplified as  2   σY 1 − ρ 2 + 1 σ X2 0 ∗ D0 = , ∗ D1∗    σ X2 1 − ρ 2 + 0 σY2 1 . = ∗  501 (42) (43) As we mentioned in the last paragraph in Sect. 3.1.1, there are two possible message orderings in the successively structured 2-terminal coding system: (a) the message from source Y is designed to be decoded first, and (b) the message from source X is designed to be decoded first. So far, we have analyzed the message ordering (a). We show that one of the corner points on the sum-rate bound of (13) is achievable with the above-mentioned coding scheme, depicted in Fig. 10. We prove this result by construction. Let   2D0 σ X2 σY2 1 − ρ 2   0 = , (44) βmax σ X2 σY2 1 − ρ 2 − 2D0 σY2 1 =   2D1 σ X2 σY2 1 − ρ 2   . βmax σ X2 σY2 1 − ρ 2 − 2D1 σ X2 Substituting (44) and (45) in (38) and (39) reveals that  2 βmax σ X2 σY2 1 − ρ 2    , R0 A = D0 βmax σY2 1 − ρ 2 − 2D1 ρ 2   βmax σY2 1 − ρ 2 − 2D1 ρ 2   . R1A = 2D1 1 − ρ 2 (45) (46) (47) which satisfy the sum-rate distortion function of (13). The achievable average distortions in (42) and (43) can be calculated as D0∗ = D0 and D1∗ = D1 . Therefore, the sum-rate distortion function of (13) is achievable. In fact, the successively structured 2-terminal coding scheme with coding/decoding of ordering (a) can achieve one corner point of the sum-rate distortion function. Analyzing decoding ordering (b) gives the same rate-distortion points as those of decoding ordering (a) in (46) and (47) with the subscripts interchanged, i.e., subscripts (0, 1, X, Y ) are replaced by (1, 0, Y, X ), respectively. As a result,   βmax σ X2 1 − ρ 2 − 2D0 ρ 2   R0B = , (48) 2D0 1 − ρ 2  2 βmax σ X2 σY2 1 − ρ 2    . (49) R1B = D1 βmax σ X2 1 − ρ 2 − 2D0 ρ 2 Therefore, the optimal rate allocation scheme that achieve any distortion pair (D0 , D1 ) using minimum sum-rate R of (13) corresponds to the points on line AB. Each corner point of this sum-rate line is achievable using one of two possible decoding orderings in the successively structured 2-terminal coding system. Allowing time-sharing between both coding schemes, i.e., the 2-terminal coding system with decoding ordering (a) and the 2-terminal coding system with decoding ordering (b), achieves all the intermediate points of line AB. ⊓ ⊔ 123 502 H. Behroozi, M. R. Soleymani 3.3 Comparison with Sequential Coding of Correlated Sources In this part, we make a comparison between the minimum sum-rate of the sequential coding [37] and the sum-rate of the successive coding strategy for coding of correlated Gaussian sources. In fact, the scenario of successive coding in the 2-terminal source coding problem is similar to the scenario of sequential coding for two correlated sources, illustrated in Sect. 1.2.3. The sequential coding scheme is depicted in Fig. 6. In this coding system, the encoding is performed sequentially, i.e., the encoding of Y depends on the source X but not vice versa. In [37], it is shown that the minimum sum-rate for the sequential coding of correlated Gaussian sources is given by     σ X2 σY2 1 D0 2 R0 + R1 = log . (50) 1−ρ 1− 2 2 D0 D1 σX Now, consider the ordering (b) of the successive coding strategy. The minimum sum-rate of the successive coding strategy for this scheme is given in (29). By rewriting (29), we observe that this is the same as the minimum sum-rate of sequential coding in (50). Therefore, there is no sum-rate loss with the successive coding compared with the sequential coding of correlated Gaussian sources. This means that the availability of side information at the encoder does not improve the sum-rate distortion function. This observation is of special interest in some applications such as video coding which is described as follows. 3.3.1 Video Coding A video source can be considered as a sequence of video frames [37,43]. Each frame of the video may be viewed as a source and a sequence of frames corresponds to a sequence of correlated sources. The video coding scenario is illustrated in Fig. 11. Video frame j, consists of a two-dimensional array of pixels, correspond to the source X j . Specifically, X j = {X j (k)}nk=1 for j = 1, 2, . . . , T represent T video frames where k denotes the spatial location of a pixel with respect to a particular spatial scan order (e.g., zig-zag), and X j (k) represents luminance intensity of the pixel at spatial location k in frame number j. This statistical structure implies that the sources are spatially independent but temporally dependent. With this perspective the video coding can be modeled as sequential encoding of correlated sources. space time Frame 1 Frame 2 Frame 3 X1 X2 X3 Fig. 11 Video coding scenario. A video source can be considered as a sequence of video frames which are spatially independent but temporally dependent. Each frame, consisting of a two-dimensional array of pixels, can be modeled as a source and a sequence of frames corresponds to a sequence of correlated sources 123 Successively Structured Gaussian Two-terminal Source Coding 503 By determining the minimum total rate R(D0 , D1 ) = R0 + R1 required to achieve distortion levels D0 and D1 in the successively structured 2-terminal source coding, we showed that there is no sum-rate loss with the successive coding compared with the sequential coding of correlated Gaussian sources. Consequently, efficient compression can be achieved by exploiting source statistics at the decoder side only. Furthermore, in most practical video codecs there are storage constraints. In particular, the encoder for encoding a frame can retain a copy of the previous frame but not earlier frames [37]. This suggests that we should consider the successive coding as a promising technique to achieve the minimum sum-rate with no storage constraint at the encoder. 4 M-terminal Source Coding In the multiterminal source coding, the FC decodes messages of all sources. The multitern minal coding system  n  with m-terminal is shown in Fig. 12. Data sequences Yi are separately encoded to ϕi Yi where the encoder functions are defined by ϕi : Yin → Ci = {1, 2, . . . , Ci } , i = 1, 2, . . . , m. (51) The coded sequences are sent to the FC with the rate constraints 1 log Ci ≤ Ri + δ, i = 1, 2, . . . , m (52) n where  δ is an arbitrary   prescribed positive number. The decoder observes the m-tuple ϕ1 Y1n , . . . , ϕm Ymn , decodes all the messages, and makes estimates of all the sources. The decoder function is given by ψ = (ψ1 , ψ2 , . . . , ψm ) where its components are defined by ψi : C1 × C2 × · · · × Cm → Yin , i = 1, 2, . . . , m. (53) i )2 . Let di : Yi2 → [0, ∞) be the squared distortion measures, i.e., di (Yi , Yi ) = (Yi − Y   n  n  n  For the reconstructed signals Yi = ψi ϕ1 Y1 , . . . , ϕm Ym , the average distortions 1 , 2 , . . . , m can be defined by   n 1 it ) . di (Yit , Y (54) i = E n t=1 n n Y1 Y1 ϕ1 ϕ1 (Y1 ) n n n Y2 Y2 ϕ2 .. . ... n Ym Ym ϕm ϕ2 (Y2 ) ψ=(ψ1 ,ψ2 ,...,ψm ) .. . Y1 .. .n Ym n ϕm(Ym) Fig. 12 m-terminal source coding scheme. Yi ’s for i = 1, . . . , m are m correlated sources that are separately encoded. The joint decoder aims to obtain estimates of all the sources 123 504 H. Behroozi, M. R. Soleymani For given distortion levels (D1 , D2 , . . . , Dm ), an m-tuple set of rates (R1 , R2 , . . . , Rm ) is admissible if for any δ > 0 and any n ≥ n 0 (δ) there exists a (m+1)−tuple (ϕ1 , ϕ2 , . . . , ϕm , ψ) satisfying (51)–(53) such that i ≤ Di + δ for i = 1, 2, . . . , m. The rate-distortion region R (D1 , D2 , . . . , Dm ) can be defined as all the m-tuple sets of rates (R1 , R2 , . . . , Rm ) that are admissible. Similar to the previous section, we first consider the special case of the m-terminal coding system, where the goal is to estimate one of these sources while other sources provide partial side information to the decoder to help reconstruction of the main source. Based on the result of the following section, we can obtain an inner region for the m-terminal coding system. 4.1 Achievable Rate Region of a Special case: m-helper Problem The m-helper system and the system based on the successive coding strategy are shown in Figs. 13 and 14, respectively. By generalizing the results of Sect. 3 and considering the fact that each decoder decodes its corresponding source, the rates of encoders can be computed as follows: 2 σY 1 log m 2 Dm 2 σ 1 Y |Z = log m−1 m 2 Dm−1 Rm = Rm−1 2 Rm−2 = .. . X ϕ0 Y1 ϕ1 .. . .. . n Ym Ym ϕ0 (X ) n n Y1 (55) n n X σY |Z ,Z 1 log m−2 m m−1 2 Dm−2 .. . ϕm ϕ1 (Y1 ) ψ0 X n n ϕm(Ym) Fig. 13 m-helper coding system. X and Yi ’s for i = 1, . . . , m are m + 1 correlated sources that are separately encoded. The joint decoder wants to obtain an estimate of the main source X while other m sources, called the helpers, play the role of partial side information to help the decoder to reconstruct the transmitted sequence of the main source 123 Successively Structured Gaussian Two-terminal Source Coding 505 Fusion Center n X X ϕ0 R0 ϕ1 R1 n Y1 Y1 Y2 ϕ2 n Ym n ~n ~ ~n ~n Ym ,Ym−1 ,..., Y2 , Y1 R2 .. . .. . Ym n Dec 1 n ~n ~ ~n Ym ,Ym−1 ,..., Y2 n Y2 X Dec 0 ϕm Dec 2 .. .~ n Ym Rm Dec m Fig. 14 Successively structured m-helper source coding scheme. By applying the successive coding/decoding strategy in the m-helper problem, the problem is decomposed into m successive coding stages 2 σY |Z ,Z ,...,Z 2 1 log 1 m m−1 2 D1 2 σ 1 X |Z m ,Z m−1 ,...,Z 2 ,Z 1 R0 = log . 2 D0   Di i where Z i = Yi = Yi + Vi , Vi ∼ N (0, D ), α = 1 − and Di is the average distortion i αi σ2 R1 = Yi in estimating source Yi at decoder i for i = 1, . . . , m. To obtain the final achievable ratedistortion region R0 (D0 ), we should obtain the conditional variances in (55) in terms of the source variances and the correlation coefficients between sources. If X i is a zero mean Gaussian random variable, X i ∼ N (0, σ X2 ), and X i ’s are jointly Gaussian, then the conditional pdf, p X n+1 |X 1 ,X 2 ,...,X n (xn+1 |x1 , x2 , . . . , xn ) can be obtained as follows [44]: 1 p X n+1 |X 1 ,X 2 ,...,X n (xn+1 |x1 , x2 , . . . , xn ) = √ 2π   (xn+1 − a1 x1 − a2 x2 − · · · − an xn )2 exp − , 2 (56) where  = σ X2 n+1 |X 1 ,X 2 ,...,X n  = Rn+1,n+1 − a1 R1,n+1 − a2 R2,n+1 − · · · − an Rn,n+1 (57) and Ri, j = E X i X j = ρ X i X j σ X i σ X j which are called the correlation functions. Coefficients a1 , a2 , . . . , an can be computed from the following equation:  E (X n+1 − a1 X 1 − a2 X 2 − · · · − an X n ) X i = 0, (58) for i = 1, 2, . . . , n. This equation can be represented as the series of equations as follows: 123 506 H. Behroozi, M. R. Soleymani ⎧ a1 R11 + a2 R21 + · · · + an Rn,1 = Rn+1,1 ⎪ ⎪ ⎪ ⎨ a1 R12 + a2 R22 + · · · + an Rn,2 = Rn+1,2 .. .. ⎪ ⎪ . . ⎪ ⎩ a1 R1,n + a2 R2,n + · · · + an Rn,n = Rn+1,n (59) To solve (59), consider it as the matrix equation AR X X = R X n+1 X where A = [a1 a2 . . . an ], X = [X 1 . . . X n ], R X X = E Xt X , and R X n+1 X = [Rn+1,1 Rn+1,2 . . . Rn+1,n ]. Therefore, the coefficient vector of A can be obtained by A = R X n+1 X R−1 XX. (60) As a result, using (57) and (60), the conditional variances required in (55) can be obtained versus the correlation functions. Since the final goal is to obtain R0 (D0 ) in terms of source variances and correlation coefficients between sources, we just need to derive the relation between   correlation coefficients ρYi Z j , ρ Z i Z j , ρ X Z i and correlation coefficients (ρ X Yi , ρYi Y j ). These coefficients can be expressed as follows: ρYi Y j  2 ρYi Z j = σY j − D j , σY j ρ X Yi  2 ρ X Zi = σYi − Di , (61) σYi  ρYi Y j  2 σYi − Di σY2j − D j . ρZi Z j = σYi σY j 4.2 Special Case: Two-helper Problem From Eq. 55, we know that  2  σY2 1 R2 (D2 ) = log , 2 D2  2  σY1 |Z 2 1 , R1 (D1 ) = log 2 D1  2  σ X |Z 1 ,Z 2 1 R0 (D0 ) = log , 2 D0 (62) where   σY21 |Z 2 = σY21 1 − ρY21 Z 2 , σ X2 |Z 1 ,Z 2 = σ X2 1 − ρ12 1 − ρ Z2 1 Z 2 (63) , ρ12 = ρ X2 Z 1 + ρ X2 Z 2 + ρ Z2 1 Z 2 − 2ρ X Z 1 ρ X Z 2 ρ Z 1 Z 2 . Combining (63) with (61) results in ⎡   ⎤ σ X2 1 − ρ12 1 ⎦, R0 (D0 ) = log ⎣ 2 D 1 − ρ2 0 123 Z1 Z2 (64) Successively Structured Gaussian Two-terminal Source Coding 507 where ρ12 and ρ Z2 1 Z 2 in terms of the correlation coefficients among X, Y1 and Y2 are as follows: ρ12 = ρ X2 Y1 + ρ X2 Y2 + ρY21 Y2 − 2ρ X Y1 ρ X Y2 ρY1 Y2   + 2ρ X Y1 ρ X Y2 ρY1 Y2 − ρ X2 Y1 − ρY21 Y2 × 1 − ρY21 Y2 + ρY21 Y2 2−2R2 2−2R1   + 2ρ X Y1 ρ X Y2 ρY1 Y2 − ρ X2 Y2 − ρY21 Y2 2−2R2   + ρY21 Y2 − 2ρ X Y1 ρ X Y2 ρY1 Y2 × 1 − ρY21 Y2 + ρY21 Y2 2−2R2 2−2R1 2−2R2 , (65) and ρ Z2 1 Z 2 = ρY21 Y2 1 − 2−2R2 × 1 − 2−2R1 1 − ρY21 Y2 + ρY21 Y2 2−2R2 . (66) Therefore, (64) defines an achievable rate-distortion region for the 2-helper problem based on the successive coding strategy. If R1 → ∞ and R2 → ∞, i.e., the case of full resolution side information, (64) can be simplified to ⎤ ⎡ σ X2 1 − ρ X2 Y1 − ρ X2 Y2 − ρY21 Y2 + 2ρ X Y1 ρ X Y2 ρY1 Y2 1 ⎦, (67) R0 (D0 ) = log ⎣ 2 D 1 − ρ2 0 Y1 Y2 which is equal to the conditional rate-distortion function, R X |Y1 ,Y2 , when both encoder and decoder have access to Y1 and Y2 . Therefore, there is no rate loss with the successive coding when compared to lossy coding of X with the side information (Y1 and Y2 ) available at both the encoder and the decoder. Example 1 Consider the two-helper problem with R2 = 0. Since the rate of the encoder 2 is zero, there is no help from Y2 . Therefore, we expect to obtain the rate-distortion for the 1-helper problem. By substituting R2 = 0 in (64), we obtain   σ X2 1 2 2 −2R1 1 − ρ X Y1 + ρ X Y1 2 R0 (D0 ) = log , (68) 2 D0 which is the rate-distortion function of the 1-helper problem. 4.3 An Inner Region for m-terminal Coding System Assume i = (πi1 , πi2 , . . . , πim ) is a permutation of the set Im = {1, 2, . . . , m}. Using the successive coding strategy, there are m! possible orderings for the coding/decoding process in the m-terminal source coding scheme. For a given permutation , the following rate region is achievable:      D(1) , D(2) , . . . , D(m) , R (D(1) , D(2) , . . . , D(m) ) = R∗(m) D(m) ∩ R (69) where R∗( j)   D( j) =   R(1) , R(2) , . . . , R(m) : R( j)  2  σY( j) |Z ( j−1) ,...,Z (1) 1 ≥ log , 2 D( j) (70) 123 508 H. Behroozi, M. R. Soleymani    D(1) , D(2) , . . . , D(m) = R m j=1 R( j) ≥ 1 2 log
)m and Z ( j) = Y( j) + V( j) , V( j) ∼ N (0,   R(1) , R(2) , . . . , R(m) : 2 j=1 σY( j) |Z ( j−1) ,...,Z (1) )m j=1 D( j) D( j) α( j) ), α( j)  = 1− * (71) , D( j) σY2 ( j)  . The inner region for the rate region of the multiterminal source coding can be represented as , , ++ m! min (D1 , D2 , . . . , Dm ) , (72) Rin (D1 , D2 , . . . , Dm ) = ∩i=1 R∗i (m) (Di (m) ) ∩ R where 1≤i≤m!    i Di (1) , Di (2) , . . . , Di (m) . min (D1 , D2 , . . . , Dm ) = min R R  (73) R∗i (m) Di (m) is in fact the rate-distortion tradeoff of the (m − 1)-helper coding scheme using the ith decoding ordering (out of m! possible orderings). 5 Conclusion In this paper, we showed that the successive coding strategy can achieve the rate-distortion function of the 1-helper problem as well as the whole rate-distortion region of the 2-terminal source coding system. Compared with the sequential coding of correlated Gaussian sources, there is no loss of sum-rate for the successive coding where the side information is not available at the encoder. This result is of special interest in some applications such as video coding, where we have processing and storage constraints at the encoder. Finally, we derived an inner bound for the rate region of the m-terminal source coding system. References 1. Gelfand, S. I., & Pinsker, M. S. (1979). Coding of sources on the basis of observations with incomplete information. Problems of Information Transmission, 15(2), 115–125. 2. Yamamoto, H., & Itoh, K. (1980). Source coding theory for multiterminal communication systems with a remote source. Transactions of IECE Japan, E63(10), 700–706. 3. Flynn, T. J., & Gray, R. M. (1987). Encoding of correlated observations. IEEE Transactions on Information Theory, 33(6), 773–787. 4. Berger, T., Zhang, Z., & Viswanathan, H. (1996). The CEO problem. IEEE Transactions on Information Theory, 42(3), 887–902. 5. Berger, T. (1978). Multiterminal source coding. In G. Longo (Ed.). The Information Theory Approach to Communications (CISM Courses and Lectures, no. 229) (pp. 171–231). Vienna, Austria: Springer-Verlag. 6. Wagner, A. B., Tavildar, S., & Viswanath, P. (2008). The rate region of the Quadratic Gaussian two-terminal source-coding problem. IEEE Transactions on Information Theory, 54(5), 1938–1961. 7. Servetto, S. D. The sum-rate of multiterminal source coding with two encoders. IEEE Transactions on Information Theory, submitted for publication, Apr. 2006 (original title: The Region of Achievable Rates for Multiterminal Source Coding, Part I). 8. Draper, S. C. (2002). Successive structuring of source coding algorithms for data fusion, buffering, and distribution in networks, Ph.D. dissertation, Cambridge, MA: Massachusetts Institute of Technology (MIT), Jun. 2002. 9. Natrio, L., Brites, C., Ascenso, J., & Pereira, F. (2005). Extrapolating side information for low-delay pixel-domain distributed video coding. Int. Workshop on Very Low Bitrate Video Coding, Sardinia, Italy, Sept. 2005. 10. Shamai, S., Verdu, S., & Zamir, R. (1998). Systematic lossy source/channel coding. IEEE Transactions on Information Theory, 44(2), 564–579. 123 Successively Structured Gaussian Two-terminal Source Coding 509 11. Girod, B., Aaron, A., Rane, S., & Rebollo-Monedero, D. (2005). Distributed video coding. Proceedings of the IEEE, 93(1), 71–83, Jan. 2005. Invited paper, Special Issue on Video Coding and Delivery. 12. Wyner, A. D., & Ziv, J. (1976). The rate-distortion function for source coding with side information at the decoder. IEEE Transactions on Information Theory, 22(1), 1–10. 13. Wyner, A. D. (1978). The rate-distortion function for source coding with side information at the decoder II: General sources. Information and Control, 38(1), 60–80. 14. Xiong, Z., Stankovic, V., Cheng, S., Liveris, A., & Sun, Y. (2004). Source-channel coding for algebraic multiterminal binning. In Proceedings on IEEE Information Theory Workshop (ITW), San Antonio, Texas, Oct. 2004. 15. Costa, M. (1983). Writing on Dirty Paper. IEEE Transactions on Information Theory, 29(3), 439–441. 16. Eswaran, K. (2006). Remote source coding and AWGN CEO problems. Master’s thesis, EECS Department, University of California, Berkeley, Jan. 2006. 17. Oohama, Y. (1997). Gaussian multiterminal source coding. IEEE Transactions on Information Theory, 43(6), 1912–1923. 18. Pandya, A., Kansal, A., Pottie, G. J., & Srivastava, M. B. (2004). Lossy source coding of multiple Gaussian sources: m-helper problem. In Proceedings on IEEE Information Theory Workshop (ITW), San Antonio, Texas, Oct. 2004. 19. Oohama, Y. (1999). Multiterminal source coding for correlated memoryless Gaussian sources with several side information at the decoder. In Proceedings on IEEE Information Theory Workshop, p. 100, Kruger National Park, South Africa, Jun. 1999. 20. Oohama, Y. (2005). Rate-distortion theory for Gaussian multiterminal source coding systems with several side informations at the decoder. IEEE Transactions on Information Theory, 51(7), 2577–2593. 21. Oohama, Y. (2006). Gaussian multiterminal source coding with several side informations at the decoder. In Proceedings on IEEE Int. Symp. Information Theory (ISIT), Seattle, Washington, Jul. 2006. 22. Berger, T., & Yeung, R. (1989). Multiterminal source encoding with one distortion criterion. IEEE Transactions on Information Theory, 35(2), 228–236. 23. Tung, S. Y. (1978). Multiterminal source coding. Ph.D. Thesis, School of Electrical Engineering, Cornell University, Ithaca, NY, May 1978. 24. Oohama, Y. (2006). Separate source coding of correlated Gaussian remote sources. In Proceedings on 2006 Information Theory and Applications Workshop, San Diego, CA, USA, Jan. 2006. 25. Koshelev, V. N. (1980). Hierarchical coding of discrete sources. Problemy Peredachi Informatsti, 16(3), 31–49. 26. Equitz, W. H. R., & Cover, T. M. (1991). Successive refinement of information. IEEE Transactions on Information Theory, 37(2), 269–275. 27. Rimoldi, B. (1994). Successive refinement of information: Characterization of achievable rates. IEEE Transactions on Information Theory, 40(1), 253–259. 28. Steinberg, Y., & Merhav, N. (2004). On successive refinement for the Wyner-Ziv problem. IEEE Transactions on Information Theory, 50(8), 1636–1654. 29. Cheng, S., & Xiong, Z. (2005). Successive refinement for the Wyner-Ziv problem and layered code design. IEEE Transactions on Signal Processing, 53(8), 3269–3281. 30. Tian, C., & Diggavi, S. (2007). On multistage successive refinement for Wyner-Ziv source coding with degraded side information. IEEE Transactions on Information Theory, 53(8), 2946–2960. 31. Viswanathan, H., & Berger, T. (1997). The Quadratic Gaussian CEO problem. IEEE Transactions on Information Theory, 43(5), 1549–1559. 32. Chen, J., Zhang, X., Berger, T., & Wicker, S. B. (2004). An upper bound on the sum-rate distortion function and its corresponding rate allocation schemes for the CEO problem. IEEE Journal on Selected Areas in Communications, 22(6), 977–987. 33. Prabhakaran, V., Tse, D., & Ramchandran, K. (2004). Rate region of the Quadratic Gaussian CEO problem. In Proc. IEEE Int. Symp. Information Theory (ISIT), June 2004, p. 119. 34. Chen, J., & Berger, T. (2008). Successive Wyner-Ziv coding scheme and its application to the Quadratic Gaussian CEO problem. IEEE Transactions on Information Theory, 54(4), 1586–1603. 35. Behroozi, H., & Soleymani, M. R. (2007). Distortion sum-rate performance of successive coding strategy in quadratic Gaussian CEO problem. IEEE Transactions on Wireless Communications, 6(12), 4361–4365. 36. Behroozi, H., & Soleymani, M. R. (2006). Successively structured Gaussian CEO problem. In Proceedings on IEEE Global Telecommunications Conference (GLOBECOM 2006), San Francisco, CA, Nov. 2006. 37. Viswanathan, H., & Berger, T. (2000). Sequential coding of correlated sources. IEEE Transactions on Information Theory, 46(1), 236–246. 123 510 H. Behroozi, M. R. Soleymani 38. Barron, R. J., Chen, B., & Wornell, G. W. (2003). The duality between information embedding and source coding with side information and some applications. IEEE Transactions on Information Theory, 49(5), 1159–1180. 39. Pradhan, S. S., Chou, J., & Ramchandran, K. (2003). Duality between source coding and channel coding and its extension to the side information case. IEEE Transactions on Information Theory, 49(5), 1181–1203. 40. Cover, T. M., & Thomas, J. A. (1991). Elements of information theory. New York: Wiley. 41. Yang, Y., Stankovic, V., Xiong, Z., & Zhao, W. (2005). On multiterminal source code design. In Proceedings on IEEE Data Compression Conference (DCC), Snowbird, Utah, Mar. 2005. 42. Yang, Y., Stankovic, V., Xiong, Z., & Zhao, W. (2008). On multiterminal source code design. IEEE Transactions on Information Theory, 54(5), 2278–2302. 43. Ma, N., Wang, Y., & Ishwar, P. (2007). Delayed sequential coding of correlated sources. 2nd Int. Workshop on Information Theory and its Applications, San Diego, CA, USA, Jan. 2007. 44. Papoulis, A. (1991). Probability, random variables, and stochastic processes. McGraw- Hill. Author Biographies Hamid Behroozi received the B.Sc. degree from University of Tehran, Tehran, Iran, the M.Sc. degree from Sharif University of Technology, Tehran, Iran, and the Ph.D. degree from Concordia University, Montreal, QC, Canada, all in Electrical Engineering in 2000, 2003, and 2007, respectively. He is currently a Postdoctoral Fellow in the Department of Mathematics and Statistics, Queen’s University. His research interests include information theory, joint source-channel coding, and cooperative communications. Dr. Behroozi is the recipient of several academic awards including Ontario Postdoctoral Fellowship awarded by the Ontario Ministry of Research and Innovation (MRI), Quebec Doctoral Research Scholarship awarded by the Government of Quebec (FQRNT), Hydro Quebec Graduate Award, and Concordia University Graduate Fellowship. M. Reza Soleymani received the B.S. degree from the University of Tehran, Tehran, Iran, in 1976, the M.S. degree from San Jose State University, San Jose, CA, in 1977, and the Ph.D. degree from Concordia University, Montreal, QC, Canada, in 1987, all in electrical engineering. From 1987 to 1990, he was an Assistant Professor with the Department of Electrical and Computer Engineering, McGill University, Montreal. From October 1990 to January 1998, he was with Spar Aerospace Ltd. (currently EMS Technologies Ltd.), Montreal, QC, Canada, where he had a leading role in the design and development of several satellite communication systems. In January 1998, he joined the Department of Electrical and Computer Engineering, Concordia University, as a Professor. His current research interests include wireless and satellite communications, information theory and coding. 123