Research Article Vol. 31, No.

2 / 16 Jan 2023 / Optics Express 3153

High-security constellation shaped

self-homodyne coherent system with 4-D joint
encryption
Y IZHAO C HEN , 1 J UNDA C HEN , 2 M INGMING Z HANG , 2 W EIHAO
L I , 2 D EMING L IU , 1,2 AND M ING TANG 2,*
1 ResearchCenter of 6 G Mobile Communications, School of Cyber Science and Engineering and Wuhan
National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, 430074,
China
2 Wuhan National Lab for Optoelectronics (WNLO) & National Engineering Laboratory for Next
Generation Internet Access System, School of Optical and Electronic Information, Huazhong University of
Science and Technology, Wuhan, 430074, China
* tangming@mail.hust.edu.cn

Abstract: In recent years, the self-homodyne coherent (SHC) system and the constellation
shaping (CS) technique have drawn considerable attention due to their abilities to further
improve the transmission capacity for various scenarios. From the security point of view, the
CS technique and the SHC infrastructure also provide new dimensions for encryption. We
propose a high-security and reliable SHC system based on the CS technique and the digital chaos.
With a four-dimensional hyperchaotic system, chaotic sequences are generated and used for the
exclusive or operation, chaotic constant composition distribution matching, phase disturbance, and
optical-layer time-delay disturbance. Moreover, 64-ary circular quadrature amplitude modulation
(64CQAM) format is adopted for transmission due to its advantages of sensitivity to phase noise,
immunity to conventional digital signal processing, and ability of time-mismatch masking, which
is verified by simulation in a SHC system. Last, we conduct an experimental verification in a
20GBaud probabilistically shaped 64CQAM SHC system. Consequently, with a large-linewidth
laser source, optical-layer security can be protected by time mismatches of tens of picoseconds.
And the digital-layer security is protected by an enormous key space of 10127 . The proposed
scheme can provide reliable real-time encryption for the optical fiber transmission, serving as a
potential candidate for the future high-capacity inter/intra-datacenter security interconnect.

© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement

1. Introduction
With the emerging of the high-end Internet applications, such as cloud computing services,
high-definition video and Internet of Things (IoT), the traffic demand is increasing sharply in the
communication network from the core to the data center level. Meanwhile, with the advances in
technology, the leading trends such as ecommerce, mobile payments and social media, bring
more cyber risk to users and businesses [1]. Thus, it can be speculated that high capacity and
security are two essential factors for the future optical communication system.
In the past, the security researches on the optical communication systems mainly focus on the
upper layer of the open system interconnection (OSI) model [2–5]. However, the upper layer
encryption cannot protect the control data and the header information, hence facing the challenge
of potential security risk [6,7]. In recent years, physical layer encryption has drawn considerable
attention, which can provide transparent encryption for all types of data [8]. There are two typical
approaches of the physical layer encryption: quantum key distribution (QKD) [9,10] and chaotic
cryptography [11–14]. To date, the development of QKD remains an ongoing challenge due to
the stability, key distribution speed, coding, digital signal processing (DSP) performance and

#477149 https://doi.org/10.1364/OE.477149
Journal © 2023 Received 3 Oct 2022; revised 14 Dec 2022; accepted 20 Dec 2022; published 12 Jan 2023
 Research Article Vol. 31, No. 2 / 16 Jan 2023 / Optics Express 3154

parallel operation [15]. As another option, chaotic cryptography has been widely investigated
in various encryption schemes due to its high initial condition sensitivity, pseudo-randomness,
and ergodicity. Chaotic encryption can be implemented in the optical domain by laser chaos
technology [16,17], or in the digital domain by DSP. Compared with the optical-domain chaos,
the digital-domain chaos using DSP can omit additional optical components and significantly
reduce system costs [18]. In [19], a physical layer security enhanced orthogonal frequency
division multiplexing (OFDM) system is proposed with three-dimensional Brownian motion and
chaos in cell. In [20], a fixed-point chaos algorithm is used to implement secure transmission
with low complexity. Besides, various digital chaos encryption schemes have been proposed
based on deep-learning [21], permutation algorithms [22,23], compressive sensing [24], and
deoxyribonucleic acid (DNA) encoding [25], etc. In terms of constellation-level operation, some
encryption schemes have been proposed, such as the chaotic constellation mapping [26], chaotic
constellation transformation [27], and two-level constellation masking [28]. Moreover, to further
enhance the security, hyperchaotic systems with more complex chaotic characteristics have been
applied in the optical communication system [8,29].
In recent years, many novel techniques have been proposed to meet the ever-increasing capacity
requirements, such as the constellation shaping (CS) technique and the self-homodyne coherent
(SHC) system [30,31]. The main concept of the CS technique is to optimize the distribution of the
constellation points, and two approaches can be distinguished. The first approach is probabilistic
shaping (PS), which employs uniformly spaced constellation points occurring with different
probabilities. The second approach is geometric shaping (GS), which employs nonuniformly
spaced constellation points. With the CS technique, non-uniform modulation formats can be
used in the transmission system, which bring the advantages of shaping gain and rate adaptivity.
On the other hand, the SHC system, which is a simplified version of the traditional coherent
system, is proposed to provide a low-cost solution for next-generation inter-/intra- datacenter
interconnects [32,33]. In the SHC system, the local oscillator (LO) shares the same laser with
the signal and is delivered remotely to the receiver for coherent reception. In this way, the impact
of the laser phase noise and the frequency offset can be omitted, allowing the usage of un-cooled
lasers with large linewidth and simpler DSP algorithms [32].
From the security point of view, these novel techniques not only play important roles in the
high-capacity transmission system, but also provide new dimensions for encryption operation.
The PS technique has been used to improve the transmissions performance of the security system.
In [34], dynamic probability chaos encryption was used in the carrier-less amplitude/phase (CAP)
system. In [35], chaotic mapping was proposed to disturb the probability mass function (PMF),
resulting in a floating distribution on the constellation. However, these schemes would change
the probability distribution, hence influencing the shaping performance of PS. To maintain the
probability distribution of PS, a novel constellation encryption scheme was proposed with two
bit-level encryption operations [36]. Taking it one step further, a chaotic constant composition
distribution matcher (CCDM) was proposed to simultaneously conduct encryption and shaping
[37]. Nevertheless, the chaotic CCDM is realized through a look up table with small blocklength,
which would lead to severe rate loss. On the other hand, some researches have emerged to
implement chaotic secure transmission in the SHC system. In [38], a hybrid optical chaos SHC
system was proposed and verified through simulation, where the system security is guaranteed
by the high dynamical complexity of the chaos source and the sensitive time delay. In [39], a
coherent secure communication scheme with a multi-core fiber (MCF) was proposed and verified
experimentally, where a phase chaotic source is deployed and the time delay between the signal
and the LO can be adjusted to guarantee the security. However, to generate the chaotic light
source, specific devices have to be introduced, which would significantly increase the system
complexity.
 Research Article Vol. 31, No. 2 / 16 Jan 2023 / Optics Express 3155

In this paper, we propose a novel PS 64-ary circular quadrature amplitude modulation (PS-
64CQAM) encryption scheme in the SHC system. With a 4-D hyperchaotic system, chaotic
encryption is introduced into both the digital layer and the optical layer. In the digital layer, 3
chaotic sequences are generated and used for exclusive or (XOR), Chaotic CCDM, and phase
disturbance, respectively. The proposed chaotic CCDM can realize PS while remaining the
probability distribution unchanged. In the optical layer, the chaotic sequence is used to disturb
the time mismatch between the signal and the LO, yielding real-time encryption and decryption.
Then, a simulation investigation is conducted, which indicates that 64CQAM outperforms
conventional 64QAM from the encryption point of view due to its sensitivity to the phase noise
and ability of the time-mismatch masking. Last, we carry out an experimental verification in a
20GBaud PS-64CQAM SHC system over 1.8 km duplexed standard single mode fiber (SSMF)
transmission. Consequently, both the optical-layer encryption and the digital-layer encryption
can effectively enhance the security of the transmission.

2. Principle
2.1. Chaotic SHC system
Figure 1 depicts the block diagram of the proposed chaotic SHC system, which includes three-step
digital-layer encryption and one-step optical-layer encryption. At the transmitter side, pseudo-
random binary sequence (PRBS) is generated as the original input signal. First, the binary signal
is processed by the XOR encryption. Second, the encrypted binary sequence is sent to the chaotic
CCDM for the PS operation and chaotic encryption. After the forward error correction (FEC)
coding, the binary sequences are mapped to a 64CQAM constellation [40]. The geometric shaped
64CQAM format enables subsequent chaotic phase disturbance and shows some benefits in the
SHC system in terms of encryption, which will be detailed in the following. Last, the encrypted
signals are modulated and transmitted through the SHC system. In the SHC system, the signal
and the LO are emitted from the same laser. Thus, when applying a broadband laser source or a
chaos laser source, the system would be very sensitive to the time mismatch between the signal
and LO. To control the path length of the signal and the LO, we deploy two tunable optical delay
lines (ODLs) at the transmitter side and the receiver side, respectively. In this way, the time delay
of the ODL can be used as the secret key.

Fig. 1. Block diagram of the proposed chaotic SHC scheme (XOR: exclusive or; CCDM:
constant composition distribution matcher; FEC: forward error correction; SHC: self-
homodyne coherent; ODL: optical delay line, Inv.: inversed).

The employed 64CQAM constellation is depicted in Fig. 2, which has been shown to achieve con-
siderable transmission reach extension compared with traditional 64QAM. More details on the con-
struction of the 64CQAM constellation can be referred in [40,41]. In the constellation, all points
are distributed on 8 circles, which take value from ρ = {1, 1.59, 1.93, 2.33, 2.58, 2.84, 3.06, 3.32}.
And there are 8 equidistant points in each circle, resulting in a total of 64 points. The labels of
 Research Article Vol. 31, No. 2 / 16 Jan 2023 / Optics Express 3156

the constellation points are marked in Fig. 2, which consist of 3 amplitude bits (labeled in red)
and 3 phase bits (labeled in black). For the mapping process, the amplitude bits determine the
radius of the symbol, and the phase bits determine the phase of the symbol. The benefit of the
64CQAM constellation in this security scheme is threefold, which will be detailed and verified in
the following.

Fig. 2. Illustration of the 64CQAM constellation.

2.2. 4-D hyperchaotic system

A 4-D hyperchaotic system is used to generate the original chaotic sequences. This system is
derived from the classical hyperchaotic Chen system [42], which has been widely used and
investigated in the optical chaotic systems [8,36,37]. The hyperchaotic system is expressed as

ẋ = a(y − x) + v
⎧
⎪
⎪
⎪
⎪
⎪
⎨ ẏ = cx − y − xz
⎪
⎪
⎪
, (1)
⎪ ż = xy − bz
⎪
⎪
⎪
⎪
⎪ v̇ = −yz + dv
⎪
⎪
⎩
where ẋ = dx/dt represents the rate of change of the system state variable x with time t. Real
constants a, b, c and d are system parameters, and x, y, z and v are state feedback controllers.
When the system parameters are set as a = 10, b = 8/3, c = 28 and d = -1, the system works
in a chaotic state. In another word, with proper initial state S0 = (x0 , y0 , z0 , v0 ), the output
sequences generated by Eq. (1) are random and non-period sequences. For the initial state
S0 = (0, 10, 20, 50), the phase diagram of the 4-D hyperchaotic system is displayed in Fig. 3. The
dynamic characteristics indicates that the system is in the chaotic state with high complication
and irregulation, promising reliable security performance for chaotic encryption.

2.3. Digital-layer encryption

The proposed chaotic SHC system includes four-step encryption operations, i.e., XOR operation,
chaotic CCDM, phase disturbance and optical path delay disturbance. Among them, the first
three encryption operations are implemented in the digital layer and the last one is deployed in
the optical layer. Correspondingly, the chaotic sequences X, Y, and Z are used for the digital-layer
encryption and the chaotic sequence V is used for the optical-layer encryption.
 Research Article Vol. 31, No. 2 / 16 Jan 2023 / Optics Express 3157

Fig. 3. Phase diagram of the 4-D hyperchaotic system.

First, the XOR operation is applied to the original binary data. To get the encryption sequences
for the XOR operation, sequence X is further processed by [36]
K1 = mod (fix(abs(X) × 1014 ), 256)>128, (2)
where function fix(·) denotes rounding to the nearest integer toward zero. Then the original data
sequence S is encrypted by the XOR operation as
S1 = xor(S, K1 ). (3)
After the XOR operation, Chaotic CCDM is applied, which operates in the PAS architecture
to transform uniform binary bits into nonuniform amplitudes. The PAS architecture for the
64CQAM mapping is shown in Fig. 4. To generate n shaped 64CQAM symbols, the binary data
are split into two tributaries: ua of length k and us of length (3-γ)n, where γ denotes the faction
of phase bits addressed by FEC overhead (OH). Then, tributary ua is fed into the CCDM, which
generates n amplitude symbols satisfying a desired distribution. These amplitudes are labelled
to 3n bits to fill the amplitude bits of the 64CQAM symbols. On the other hand, a systematic
encoder copies us together with 3n amplitude bits to the final codeword and appends γn parity bits.
As these parity bits satisfy uniform distribution, they can be multiplexed with us to constitute the
phase bits [30]. Last, the amplitude bits and the phase bits are concatenated and mapped into the
PS 64CQAM signals. To maximum the shaping gain, the widely-deployed Maxwell-Boltzmann
(MB) distribution is desired for the output of CCDM, which can be expressed as
8
2 ∑︂ 2
P(ri ) = e−u |ri | / e−u |rk | , (4)
k=1

where u is the shaping coefficient and ri is the amplitude which satisfies ri ∈ ρ. The basic
principle of CCDM is to pre-define the compositions of the amplitudes in the output sequence,
and then map the uniform binary bits into an arrangement of these compositions [43]. With the
blocklength of N, the compositions in the output can be calculated by
n(ri ) ≈ P(ri ) · N. (5)
As all n(ri ) should be integers and add up to N, there are multiple possibilities for their values.
This issue is solved by adjusting the allocation as
P̄(ri ) = argmin D(P̄(ri )||P(ri )||) , subject to P̄(ri ) is N - type, (6)
where N-type means a type based on sequence of length N [43]. Thus, the compositions are
finally calculated to be
n(ri ) = P̄(ri ) · N. (7)
To further enhance the security of the transmission, we introduce the chaotic encryption
into the CCDM mapping process, resulting in a chaotic CCDM. The schematic diagram of the
 Research Article Vol. 31, No. 2 / 16 Jan 2023 / Optics Express 3158

Fig. 4. The PAS architecture for the 64CQAM mapping.

chaotic CCDM is shown in Fig. 5, in which the total blocklength is set to be 100 as an example.
Different colors are used to represent 8 amplitude compositions of the 64CQAM constellation. To
approach the MB distribution with the shaping coefficient of u = 0.22, the output compositions
are calculated by Eq. (7) to be n(r) = (30, 21, 16, 11, 8, 6, 5, 3). First, these compositions are
sorted in order as shown in Fig. 5(b). Second, the chaotic sequence Y is used for the chaotic
CCDM and pre-processed by

K2 = mod (abs(Y) × 1010 , 1). (8)

Fig. 5. The schematic diagram of the chaotic CCDM with blocklength of 100 and shaping
coefficient of 0.8. (a) Composition of the amplitudes in the block. (b) Initial ordering of
the amplitudes. (c) Chaotic sequence K 2 with length of 100. (d) The shaping process for
CCDM with resultant composition. (e) The output of CCDM. (f) The final output of the
chaotic CCDM process.

Figure 5(c) depicts the sequence K 2 of length 100. Then, 5 points are chosen out by sorting
the sequence K 2 . These points are mapped to the composition sequence in Fig. 5(b). In this
 Research Article Vol. 31, No. 2 / 16 Jan 2023 / Optics Express 3159

way, 5 compositions can be selected, whose values and positions will be fixed in the final output
sequence. For the given example, there are 2 r1 , 2 r2 , and 1 r4 chosen to be fixed, resulting in
the remaining compositions of n(r) = (28, 19, 16, 10, 8, 6, 5, 3). The remaining compositions
are used as the parameter for CCDM, as depicted in Fig. 5(d). In the shaping process, CCDM
rearranges these compositions according to the input binary data. Last, together with the 5 fixed
points, the output of CCDM of length 95 is filled into the remaining positions, yielding the final
output, as depicted in Fig. 5(e).
After the chaotic CCDM and the 64CQAM mapping, phase disturbance is applied. Due to the
rotational symmetry of the 64CQAM constellation, constellation points with different radii can
be disturbed together. Chaotic sequence Z is used, which is pre-processed by

K3 = mod (fix(abs(Z) × 1014 ), 8). (9)

Then, the symbol sequence S2 are encrypted by

S3 = S2 × ejπK3 /8 . (10)

Namely, each point in the constellation is rotated by an angle of multiples of π/8 according to
the chaotic sequence K3 , as illustrated in Fig. 6. Note that the dot size represents the probability
of different constellation points after PS. Consequently, the chaotic phase disturbance process
would not change the geometric distribution and probability distribution of the constellation,
resulting in no performance degradation.

Fig. 6. Schematic diagram of the chaotic phase disturbance for the 64CQAM constellation,
where the dot size represents the probability of different constellation points after PS. Two
colors are used to distinguish the constellation points on adjacent circles. And the gray and
black dots represent symbols before and after phase disturbance, respectively.

2.4. Optical-layer encryption

The optical-layer encryption is implemented in the SHC system, as depicted in Fig. 7. The laser
from the source is split into two tributaries, of which one is used for the signal modulation, and
the other one is used as the LO and delivered remotely to the coherent receiver. Thus, the signal
and LO share the same phase fluctuations. If the optical paths of the signal and LO are matched
well, the phase noise would be negligible for the received signal. On the contrary, if a chaotic
laser or a large-linewidth laser is used, a tiny time mismatch would lead to severe phase noise.
Thus, the time mismatch between the signal and the LO can be used as the secret key to realize
physical-layer encryption [38,39]. The probability density function (PDF) of the phase noise can
 Research Article Vol. 31, No. 2 / 16 Jan 2023 / Optics Express 3160

be obtained by [44]
θpn 2
1 −
fpn (θ pn ) = √︁ e 4π∆v·|τ∆L | , (11)
2π ∆v|τ∆L |
where ∆v and τ∆L denote the laser linewidth and the time mismatch, respectively. Figure 8
depicts the PDFs of phase noise for various time mismatches with 3 GHz laser linewidth. For the
time mismatch beyond 10 ps, the phase noise could easily exceed π/8, resulting in irrecoverable
impairment on the 64CQAM signal. In the security SHC system, two ODLs are deployed at the
transmitter side (in the signal path) and the receiver side (in the LO path), respectively. The
chaotic sequence V is used to adjust the time delays of the ODLs for different time intervals,
which is processed by

∆t = ∆T [mod (fix(abs(V) × 1014 ), L2 + 1)], (12)

where ∆T denotes the time interval and L 2 is the modulus of the set for ∆t. In this way, the
ODL of the legal receiver can keep the same time delay with that of the transmitter, resulting
in negligible phase noise. However, for the illegal receiver without the chaotic sequence, the
received signal would suffer from severe phase noise, which would severely deteriorate the
transmission performance.

Fig. 7. Block diagram of the chaotic SHC system. (DSP: digital signal processing; AWG:
arbitrary waveform generator; DP-IQM: dual-polarization IQ modulator; ODL: optical delay
line).

Fig. 8. Probability density function of phase noise for the SHC system with 3 GHz linewidth
and different time mismatch τ∆L .

In summary, the proposed chaotic SHC system includes four layers of chaotic encryption,
which are controlled by a 4-D hyperchaotic system. The first three layers are deployed in the
digital domain and is reversible, so legal receivers can generate the corresponding decryption
factor according to the key and decrypt the original information. The last layer is deployed in
the optical domain, where legal receivers have to receive the optical signal and the LO with
correct time delay according to the key. Moreover, even if the signal is received and stored by
 Research Article Vol. 31, No. 2 / 16 Jan 2023 / Optics Express 3161

illegal receivers with wrong time delay, the digital decryption can never solve the irrecoverable
phase-noise impairment, yielding real-time encryption to the signal.

3. Simulation
A simulation is carried out to investigate the performance of the 64CQAM format in the SHC
system via commercial software VPI Transmissionmaker 9.9. The simulation setup is shown
in Fig. 9. Offline signals of 64CQAM and 64QAM are generated and loaded to the simulation
system at the rate of 20 GBaud. The linewidth of the laser is set to be 1 GHz. First, the light
from the laser is equally split into two tributaries. One tributary is sent to the dual-polarization
In-phase quadrature modulator (DP-IQM) for optical modulation, and the other tributary is used
as the LO. Second, the optical signal and LO are transmitted over 2-km standard signal-mode
fiber (SSMF), respectively. Afterwards, a module is deployed in the signal path to adjust the
optical signal-to-noise ratio (OSNR) of the signal. On the other hand, a delay module is deployed
in the LO path to introduce the time mismatch between the signal and the LO, and a polarization
controller (PC) is used to stabilize the state of polarization (SOP) of the LO. Last, the signal and
the LO are mixed and received by a coherent receiver. The received digital signals are processed
by DSP, which consists of resampling, matched filter, radius-directed constant modulus algorithm
(RD-CMA), blind phase search (BPS), decision-directed least mean square (DD-LMS), and
decision.

Fig. 9. The simulation setup for the investigation of the 64CQAM format in the SHC
system (DP-IQM: dual-polarization IQ modulator; OSNR: optical signal-to-noise ratio; PC:
polarization controller; RD-CMA: radius-directed constant modulus algorithm; BPS: blind
phase search; DD-LMS: decision-directed least mean square).

For comparison, the OSNRs for the 64CQAM format and the conventional 64QAM format
are set to be 27 dB and 27.3 dB, respectively, hence resulting in identical bit error rate (BER)
performance in the context of none time mismatch. By scanning the time mismatch τ from
0 ps to 15 ps, the BER versus τ for 64CQAM and 64QAM is shown in Fig. 10(a). And the
received constellations after DSP for time mismatch of 0 ps, 5 ps, 10 ps, and 15 ps are depicted
in Fig. 10 (b). In general, the BER performance of the 64CQAM and 64QAM transmission
degrade sharply with the increase of the time mismatch. Compared with 64QAM, 64CQAM
behaves more sensitive to the phase noise induced by the time mismatch. This could be owing to
the smaller phase differences between the adjacent points of the 64CQAM constellation.
Moreover, the received constellations before DSP for 64CQAM and 64QAM with various
time mismatches are depicted in Fig. 11 in terms of the density map. It can be seen that the
density maps of the 64QAM constellations with time mismatches from 0 ps to 20 ps shows
square shape on their envelopes. On the other hand, for the 64CQAM constellations, the density
maps stay in the circle shape with the time mismatches from 0 ps to 60 ps. From the encryption
point of view, using the 64QAM format in the chaotic SHC system faces the risk of leaking
the time-mismatch information, because the illegal receiver can estimate the time mismatch by
monitoring the constellation shapes before DSP. In this regard, the 64CQAM format can promise
good encryption performance on the time mismatch. Thus, we can conclude that the benefits of
the 64CQAM format are threefold. First, the 64CQAM format can defend the brute force of the
 Research Article Vol. 31, No. 2 / 16 Jan 2023 / Optics Express 3162

Fig. 10. (a) BER vs. time mismatch for 64QAM and 64CQAM. (b) The received
constellations after DSP for 64QAM and 64CQAM with various time mismatches.

illegal receiver with traditional DSP, because the corresponding RD-CMA, BPS, and DDLMS
algorithms have to be specially designed. Second, the 64CQAM constellation is more sensitive to
the time mismatch induced phase noise, yielding better performance of the time-delay encryption.
Third, the 64CQAM format can mask the time-mismatch information with a circular shape on
the constellation density map. These benefits indicate that 64CQAM is a better choice in the
proposed chaotic SHC system.

Fig. 11. The density maps of the received constellations before DSP for 64QAM and
64CQAM with various time mismatches.

4. Experimental results
4.1. Experimental setup
The proof-of-concept experimental setup of the chaotic SHC system and the corresponding
DSP flows are shown in Fig. 12(a). First, a distributed feedback (DFB) laser (Yenista, ∼3-MHz
linewidth) is used as the optical source, which works at the wavelength of 1549.34 nm with
14.6-dBm power. To obtain a large-linewidth laser, the optical phase disturbance is introduced
by a phase modulator (PM, IXblue MPZ-LN-10), which is driven by random electrical signals
from an arbitrary waveform generator (AWG, Keysight M8195A). The disturbance rates of the
electrical signals are chosen to be 0.5 GHz, 1 GHz, 3 GHz, and 5 GHz in the experiment. And
the resultant power spectra of the disturbed laser are monitored by an optical spectrum analyzer
(OSA, Apex 2081A) and depicted in Fig. 12(b). Then, the laser is split into two tributaries for the
SHC transmission. One tributary is sent to a dual-polarization IQ modulator (DP-IQM, Fujitsu
FTM7977HQA, 23-GHz bandwidth) for modulation, and the other tributary is used as the LO.
The modulated signal is generated by offline DSP, which includes chaotic CCDM, 64CQAM
mapping, phase disturbance and root raised cosine (RRC) filter. The blocklength and the shaping
 Research Article Vol. 31, No. 2 / 16 Jan 2023 / Optics Express 3163

rate of CCDM is chosen to be 10000 and 1.5 bits/amp, respectively. Namely, for each block, 15000
binary data are mapped to 10000 amplitudes by chaotic CCDM, and finally resulting in 30000
bits to fill the amplitude bits of the 64CQAM symbols. The encrypted 20-Gaud PS-64CQAM
signal is resampled to 90 GSa/s to fit the sampling speed range of the AWG (Keysight M8196A),
and then modulated to the optical carrier by the DP-IQM. Subsequently, the optical signal and
the LO are transmitted over 1.8-km duplex standard single mode fiber. For the signal path, an
amplified spontaneous emission (ASE) source and an OSA (Yokogawa AQ6370C) are deployed
to adjust and monitor the received OSNR, respectively. Then an ODL is used to introduce time
mismatch between the signal and the LO. For the LO path, a polarization controller is deployed to
stabilize the SOP of the LO. Last, the signal is received by an integrated coherent receiver (ICR,
NeoPhotonics Class 40), and captured by a digital storage oscilloscope (DSO, Lecroy-10-36Zi) for
offline DSP. The receiver-side DSP includes Gram-Schmidt orthogonalization procedure (GSOP),
clock recovery, matched filter, radius-directed constant modulus algorithm (RD-CMA), blind
phase search (BPS), decision-directed least mean square (DD-LMS), decision and decryption.

Fig. 12. (a) Experimental setup of the chaotic SHC system and the corresponding DSP
flows (DFB: distributed feedback laser; AWG: arbitrary waveform generator; PM: phase
modulator; OSA: optical spectrum analyzer; ASE: amplified spontaneous emission; ODL:
optical delay line; PC: polarization controller; ICR: integrated coherent receiver; DSO:
digital storage oscilloscope; DSP: digital signal processing; RRC: root raised cosine; GSOP:
Gram-Schmidt orthogonalization procedure; RD-CMA: radius-directed constant modulus
algorithm; BPS: blind phase search; DD-LMS: decision-directed least mean square). (b)
Power spectra of the laser after the phase disturbance.

4.2. Results and discussion

First, we verify the encryption performance of the time mismatch disturbance in the SHC system.
As the optical path and the LO path are matched well in advance, time mismatch can be adjusted
by introducing extra time delay with the ODL. Figure 13(a) shows the experimental results
of BER versus time mismatch for various disturbance rates, for which the optical spectra are
shown in Fig. 12(b). In general, with the increase of disturbance rate, the BER performance of
the system becomes more sensitive to the time mismatch. For the disturbance rate of 3 GHz,
time mismatch higher than 60 ps can lead to significant phase noise. As for the disturbance
 Research Article Vol. 31, No. 2 / 16 Jan 2023 / Optics Express 3164

rate of 5 GHz, the time-mismatch sensitivity of the system achieves 30 ps. Consequently, for
the practical implementation, chaotic disturbance of step ∆T>60ps and ∆T>30ps can promise
secure transmission for the disturbance rate of 3 GHz and 5 GHz, respectively. Moreover, we
investigate the performance of the BPS algorithm for the mismatch induced phase noise, as
shown in Fig. 13(b). It can be seen that the BPS algorithm can slightly alleviate the impact of the
phase noise for small time mismatches. However, when the time mismatch gets larger (>30 ps for
5-GHz disturbance rate and >60 ps for 3-GHz disturbance rate), the phase noise is too severe to
be compensated for by the BPS algorithm. Consequently, the security of the transmission can be
promised by the time mismatch disturbance, because large time mismatch induced phase noise
would severely deteriorate the transmission performance and cannot be compensated for by DSP.
Even if the secret key is leaked afterwards, the signal received with wrong time delay cannot be
recovered with any digital method, yielding real-time encryption for the security transmission.

Fig. 13. (a) BER vs. time mismatch for various disturbance rates. (b) Investigation on the
performance of the BPS algorithm for mismatch induced phase noise.

Second, we investigate the performance of the digital-layer encryption. The BER versus
ONSR for various time mismatches are shown in Fig. 14(a). For the time mismatch of 0 ps,
increasing the OSNR can effectively improve the BER performance. As for the time mismatches
of 20 ps and 40 ps, the effect of increasing the OSNR is very limited. When the time mismatch
reaches 80 ps, little improvement is achieved by increasing the OSNR. On the other hand, for the
illegal receiver with wrong original key, the received signal cannot be demodulated, resulting
in the BERs around 0.5. Figure 14(b) further depicts the BER curves of an illegal receiver in
the configurations that any one chaotic sequence out of {X, Y, Z} is unknow. It can be seen
that, incorrect sequences X, Y and Z lead to deteriorated BERs around 0.5, 0.167 and 0.33,
respectively. Note that incorrect sequence Y for the receiver only influences the binary output
of CCDM. As the shaping rate of the CCDM is 1.5 bits/amp, yielding 1.5 original bits for one
symbol. Assuming that there are no error phase bits, the statistical BER should be approximately
1/3*1/2 = 1/6, which is in accordance with the experimental results. Similarly, incorrect sequence
Z only influences the phase bits, yielding statistical BER of approximately 2/3*1/2 = 1/3.
Last, the sensitivity of the key is investigated by applying a tiny change on the initial state
(x0 , y0 , z0 , v0 ) and the parameters (a, b, c, d), as shown in Fig. 15(a) and Fig. 15(b), respectively.
Consequently, for the initial state value x0 , y0 , z0 and v0 , tiny changes of up to 1e-17, 1e-16, 1e-15
and 1e-15 would change the chaotic system, respectively. As for the parameters a, b, c, and d, tiny
changes of up to 1e-16, 1e-16, 1e-15 and 1e-17 would change the chaotic system, respectively.
These tine changes would lead to completely different chaotic sequences for decryption, resulting
in sharp deterioration on the BER performance. The key space of the chaotic system can be
conservatively estimated as (1015 )3 × (1016 )3 × (1017 )2 = 10127 , which is enough to prevent the
brute-force attack of the illegal receivers.
 Research Article Vol. 31, No. 2 / 16 Jan 2023 / Optics Express 3165

Fig. 14. (a) BER vs. OSNR for legal and illegal receivers with various delay mismatches.
(b) BER vs. OSNR for illegal receiver with an unknow chaotic sequence.

Fig. 15. BER curves with a tiny absolute deviation on the (a) initial state and (b) system
parameters.

5. Conclusion
We propose a novel constellation shaping based encryption scheme in the SHC system. With
a 4-D hyper-chaotic system, 4 steps of encryption operations are applied in both the digital
layer and the optical layer. In the digital layer, we introduce the chaotic encryption to the
shaping process of the CCDM, yielding a chaotic CCDM when maintaining the probability
distribution. Geometric shaped 64CQAM format is used for transmission due to its advantages
of sensitivity to the phase noise, immunity to conventional DSP, and ability of time-mismatch
masking. Optical-layer encryption is realized by introducing disturbed time mismatch between
the signal and the LO in the SHC system, achieving real-time encryption and decryption of
the signal. To verify the proposed encryption scheme, we carry out a 20GBaud PS-64CQAM
transmission over 1.8 km duplexed SSMF in a SHC system. The experimental results indicate
that both the optical-layer encryption and the digital-layer encryption can effectively enhance the
security of the transmission. The 4-D hyper-chaotic system provides a conservative key space
of 10127 to prevent brute-force attacks. The proposed scheme can provide reliable digital-layer
encryption and real-time optical-layer encryption, serving as a potential candidate for the future
high-capacity inter/intra datacenter security interconnect.
Funding. National Natural Science Foundation of China (62225110).
Disclosures. The authors declare no conflicts of interest.
 Research Article Vol. 31, No. 2 / 16 Jan 2023 / Optics Express 3166

Data availability. Data underlying the results presented in this paper are not publicly available at this time but may
be obtained from the authors upon reasonable request.

References
1. Cisco white paper, “Cisco annual Internet report, 2018–2023”
https://www.cisco.com/c/en/us/solutions/collateral/executive-perspectives/annual-internet-report/white-paper-c11-
741490.pdf (Cisco, 2020).
2. M. Hossen, K. D. Kim, and Y. Park, “Synchronized latency secured MAC protocol for PON based large sensor
network,” in Proc. 12th Int. Conf. Adv. Commun. Technol., 1528–1532 (2010).
3. M. C. Yuang, P. Tien, D. Hsu, S. Chen, C. Wei, J. Shih, and J. Chen, “A high-performance OFDMA-PON system
architecture and medium access control,” J. Lightwave Technol. 30(11), 1685–1693 (2012).
4. Z. Zhu, W. Lu, L. Zhang, and N. Ansari, “Dynamic service provisioning in elastic optical networks with hybrid
single-/multi-path routing,” J. Lightwave Technol. 31(1), 15–22 (2013).
5. L. Gong and Z. Zhu, “Virtual optical network embedding (VONE) over elastic optical networks,” J. Lightwave
Technol. 32(3), 450–460 (2014).
6. M. Bi, X. Fu, X. Zhou, L. Zhang, G. Yang, X. Yang, S. Xiao, and W. Hu, “A key space enhanced chaotic encryption
scheme for physical layer security in ofdm-pon,” IEEE Photonics J. 9(1), 1–10 (2017).
7. K. Zhang, J. Zhang, G. Gao, and A. Fei, “Physical layer security based on chaotic spatial symbol transforming in
fiber-optic systems,” IEEE Photonics J. 10(6), 1–9 (2018).
8. J. Zhao, B. Liu, Y. Mao, R. Ullah, J. Ren, S. Chen, L. Jiang, S. Han, J. Zhang, and J. Shen, “High security OFDM-PON
with a physical layer encryption based on 4D-hyperchaos and dimension coordination optimization,” Opt. Express
28(14), 21236 (2020).
9. P. Shor and J. Preskill, “Simple proof of security of the BB84 quantum key distribution protocol,” Phys. Rev. Lett.
85(2), 441–444 (2000).
10. N. Gisin, G. Ribordy, W. Tittel, and H. Zbinden, “Quantum cryptography,” Rev. Mod. Phys. 74(1), 145–195 (2002).
11. G. D. van Wiggeren and R. Roy, “Communication with chaotic lasers,” Science 279(5354), 1198–1200 (1998).
12. A. Argyris, D. Syvridis, L. Larger, V. Annovazzi-Lodi, P. Colet, I. Fischer, J. García-Ojalvo, C. R. Mirasso, L.
Pesquera, and K. Alan Shore, “Chaos-based communications at high bit rates using commercial fibre-optic links,”
Nature 438(7066), 343–346 (2005).
13. L. J. Zhang, B. Liu, X. J. Xin, Q. Zhang, J. J. Yu, and Y. J. Wang, “Theory and performance analyses in secure
CO-OFDM transmission system based on two-dimensional permutation,” J. Lightwave Technol. 31(1), 74–80 (2013).
14. L. J. Zhang, B. Liu, and X. J. Xin, “Secure coherent optical multi-carrier system with four-dimensional modulation
space and Stokes vector scrambling,” Opt. Lett. 40(12), 2858–2861 (2015).
15. T. A. Eriksson, R. S. Luís, G. Rademacher, B. J. Puttnam, K. Gumüş, L. Schmalen, A. Alvarado, H. Furukawa, N.
Wada, T. Hirano, M. Sasaki, and M. Takeoka, “Challenges in coding, DSP and parallel operation of quantum key
distribution and coherent data transmission,” Proc. Eur. Conf. Optical Commun. (ECOC), 1–4 (2020).
16. X. Jiang, D. Liu, M. Cheng, L. Deng, S. Fu, M. Zhang, M. Tang, and P. Shum, “High-frequency reverse-time chaos
generation using an optical matched filter,” Opt. Lett. 41(6), 1157–1160 (2016).
17. Z. Zhao, M. Cheng, C. Luo, L. Deng, M. Zhang, S. Fu, M. Tang, P. Shum, and D. Liu, “Semiconductor-laser-based
hybrid chaos source and its application in secure key distribution,” Opt. Lett. 44(10), 2605–2608 (2019).
18. R. Tang, J. Ren, J. Fang, Y. Mao, Y. Han, J. Shen, Q. Zhong, X. Wu, F. Tian, and B. Liu, “Security strategy of parallel
bit interleaved FBMC/OQAM based on four-dimensional chaos,” Opt. Express 29(15), 24561–24575 (2021).
19. T. Wu, C. Zhang, C. Chen, H. Hou, H. Wei, S. Hu, and K. Qiu, “Security enhancement for OFDM-PON using
Brownian motion and chaos in cell,” Opt. Express 26(18), 22857–22865 (2018).
20. S. Li, M. Cheng, L. Deng, S. Fu, M. Zhang, M. Tang, P. Shum, and D. Liu, “Secure strategy for OFDM-PON using
digital chaos algorithm with fixed-point implementation,” J. Lightwave Technol. 36(20), 4826–4833 (2018).
21. J. Ke, L. Yi, Z. Yang, Y. Yang, Q. Zhuge, Y. Chen, and W. Hu, “32 Gb/s chaotic optical communications by
deep-learning-based chaos synchronization,” Opt. Lett. 44(23), 5776–5779 (2019).
22. B. Zhu, F. Wang, and J. Yu, “A chaotic encryption scheme in DMT for IM/DD intra-datacenter interconnects,” IEEE
Photon. Technol. Lett. 33(8), 383–386 (2021).
23. Y. Chen, Y. Huang, J. Fu, Y. Han, K. Li, and J. Yu, “Multi wings chaotic encryption scheme for PAM-DMT-based
optical access network,” IEEE Photonics J. 13(6), 1–6 (2021).
24. S. Kayalvizhi and S. Malarvizhi, “A novel encrypted compressive sensing of images based on fractional order hyper
chaotic Chen system and DNA operations,” Multimed. Tools. Appl. 79(5-6), 3957–3974 (2020).
25. M. Babaei, “A novel text and image encryption method based on chaos theory and DNA computing,” Nat. Comput.
12(1), 101–107 (2013).
26. A. Sultan, X. Yang, A. A. E. Hajomer, and W. Hu, “Chaotic constellation mapping for physical-layer data encryption
in OFDM-PON,” IEEE Photon. Technol. Lett. 30(4), 339–342 (2018).
27. W. Zhang, C. Zhang, C. Chen, and K. Qiu, “Experimental demonstration of security-enhanced OFDMA-PON using
chaotic constellation transformation and pilot-aided secure key agreement,” J. Lightwave Technol. 35(9), 1524–1530
(2017).
28. S. Chen, B. Liu, Y. Mao, J. Ren, X. Song, R. Ullah, D. Zhao, L. Jiang, S. Han, J. Zhao, J. Shen, and X. Liu, “Physical
layer data encryption using two-level constellation masking in 3D-CAP-PON,” Chin. Opt. Lett. 19(1), 010601 (2021).
 Research Article Vol. 31, No. 2 / 16 Jan 2023 / Optics Express 3167

29. M. Li, B. Liu, R. Ullah, J. Ren, Y. Mao, S. Han, J. Zhao, R. Tang, S. Chen, and J. Ling, “5D data iteration in a
multi-wavelength OFDM-PON using the hyperchaotic system,” Opt. Lett. 45(17), 4960–4963 (2020).
30. G. Böcherer, F. Steiner, and P. Schulte, “Bandwidth efficient and rate-matched low-density parity-check coded
modulation,” IEEE Trans. Commun. 63(12), 4651–4665 (2015).
31. M. Morsy-Osman, M. Sowailem, E. El-Fiky, T. Goodwill, T. Hoang, S. Lessard, and D. V. Plant, “DSP-free‘coherent-
lite’ transceiver for next generation single wavelength optical intra-datacenter interconnects,” Opt. Express 26(7),
8890–8903 (2018).
32. T. Gui, X. Wang, M. Tang, Y. Yu, Y. Lu, and L. Li, “Real-time demonstration of homodyne coherent bidirectional
transmission for next-generation data center interconnects,” J. Lightwave Technol. 39(4), 1231–1238 (2021).
33. H. Ji, X. Zhou, C. Sun, and W. Shieh, “Polarization-diversity receiver using remotely delivered local oscillator
without optical polarization control,” Opt. Express 28(15), 22882–22890 (2020).
34. S. Han, B. Liu, Y. Mao, J. Ren, X. Wu, X. Xu, L. Jiang, J. Zhang, and J. Zhao, “High-security multi-slot chaos
encryption with dynamic probability for 16-CAP PON,” IEEE Photonics J. 12(3), 1–10 (2020).
35. J. Zhao, B. Liu, Y. Mao, J. Ren, X. Xu, X. Wu, L. Jiang, S. Han, and J. Zhang, “High-security physical layer in
CAP-PON system based on floating probability disturbance,” IEEE Photon. Technol. Lett. 32(7), 367–370 (2020).
36. Z. Wang, Y. Xiao, S. Wang, Y. Yan, B. Wang, Y. Chen, Z. Zhou, J. He, and L. Yang, “Probabilistic shaping based
constellation encryption for physical layer security in OFDM RoF system,” Opt. Express 29(12), 17890–17901
(2021).
37. J. Ren, B. Liu, D. Zhao, S. Han, S. Chen, Y. Mao, Y. Wu, X. Song, J. Zhao, X. Liu, and X. Xin, “Chaotic
constant composition distribution matching for physical layer security in a PS-OFDM-PON,” Opt. Express 28(26),
39266–39276 (2020).
38. Y. Fu, M. Cheng, W. Shao, H. Luo, D. Li, L. Deng, Q. Yang, and D. Liu, “Analog-digital hybrid chaos-based
long-haul coherent optical secure communication,” Opt. Lett. 46(7), 1506–1509 (2021).
39. Y. Wu, H. Luo, M. Cheng, C. Huang, Z. Zhang, L. Deng, Q. Yang, M. Tang, X. Dai, and D. Liu, “Capacity expansion
of chaotic secure transmission system based on coherent optical detection and space division multiplexing over
multi-core fiber,” Opt. Lett. 47(3), 726–729 (2022).
40. F. Jardel, T. A. Eriksson, F. Buchali, W. Idler, A. Ghazisaeidi, C. Measson, and J. J. Boutros, “Experimental
comparison of 64QAM and combined geometric-probabilistic shaped 64QAM,” Proc. Eur. Conf. Optical Commun.
(ECOC), 1–4 (2017).
41. J. J. Boutros, F. Jardel, and C. Méasson, “Probabilistic shaping and non-binary codes,” Proc. IEEE International
Symposium on Information Theory (ISIT), (2308-2312) 2017.
42. Y. X. Li, W. K. S. Tang, and G. R. Chen, “Generating hyperchaos via state feedback control,” Int. J. Bifurcation
Chaos 15(10), 3367–3375 (2005).
43. P. Schulte and G. Bocherer, “Constant composition distribution matching,” IEEE Trans. Inform. Theory 62(1),
430–434 (2016).
44. X. Zhou, Y. Gao, J. Huo, and W. Shieh, “Theoretical analysis of phase noise induced by laser linewidth and mismatch
length in self-homodyne coherent systems,” J. Lightwave Technol. 39(5), 1312–1321 (2021).