Authors:    Siqi Chen^1,a*   ，Zijie Ma^2,b

 Affiliations:

¹SCHOOL OF PHYSICS & OPTOELECTRONIC ENGINEERING, guangdong university of technology, Guangzhou 510006, Guangdong, China

²SCHOOL OF PHYSICS & OPTOELECTRONIC ENGINEERING, guangdong university of technology, Guangzhou 510006, Guangdong, China

 

All authors’ email:

^aEmail: 3223006671@mail2.gdut.edu.cn

^bEmail: 3070947685@qq.com

Abstract: With the increasingly prominent complex channel effects such as atmospheric turbulence and multi-mode coupling of optical fibers, the reliability of the physical layer of optical communication is facing unprecedented challenges. However, traditional adaptive optics and pure data-driven compensation still face limitations in terms of physical consistency, latency bottlenecks, and heterogeneous hardware collaboration, making it difficult to balance the requirements of high precision and microsecond level response. To this end, this paper proposes a deep neural network adaptive optics (DNN-AO) fusion architecture with physical constraint embedding, which not only embeds high-order physical constraints to ensure consistency, but also balances accuracy and real-time performance through tensor quantization channel representation and dual stream attention mechanism. A parallel pipeline and ping-pong buffer collaborative system is constructed on FPGA, DSP, and Edge TPU heterogeneous platforms. The experimental results show that this scheme achieves performance leaps compared to SPGD and pure DNN baselines in nine key indicators, including RMS, PVE, Strehl ratio, polarization degradation, and Q factor. The fusion framework proposed by this research institute not only expands the collaborative paradigm of complex channels in theory, but also lays a solid foundation for reliable transmission in future space in engineering practice.

Keywords: Complex channel; Optical communication physical layer; Deep neural network; Adaptive optical compensation

I. Introduction

Optical communication, as the core support of modern information infrastructure, directly determines the capacity, speed, and reliability of information transmission through its physical layer performance. It is the strategic cornerstone for promoting the evolution of communication technology from 5G to 6G and supporting the construction of integrated air space ground networks (Kaushal & Kaddoum, 2016). Therefore, in the wave of the information age, optical communication serves as the core pillar supporting massive data transmission, and its technological innovation is directly related to the strategic layout of national digital infrastructure (Agrell et al., 2024). However, multiple challenges in complex channel environments, such as beam drift caused by atmospheric turbulence, dispersion and nonlinear effects in fiber optic transmission, continue to constrain the transmission efficiency of the optical communication physical layer, posing a severe challenge to the high-quality transmission of real-time signals (Kaushal & Kaddoum, 2016; Raj et al., 2023). In recent years, with the development of 6G ultra wideband photon terahertz real-time transmission systems and breakthroughs in optical link technology in quantum key distribution, the application requirements of optical communication in complex environments have become more prominent, and the limitations of traditional technology systems urgently need to be overcome (Raj et al., 2023). Research in related fields is also emerging like mushrooms after rain. As shown in Figure 1, searching for keywords such as “optical communication”, “optical communication physical layer”, “machine learning”, “optical communication transmission”, and “optical communication technology” on authoritative literature retrieval platforms such as Web of Science and CNKI reveals that the number of related papers has been continuously increasing in recent years.

Figure 1 Trend Chart of Paper Publication

Frontier research has made significant progress in the direction of technological integration. The wide area wavefront computing sensing chip WISE developed by the Tsinghua University team has achieved real-time detection of atmospheric turbulence in the range of over 1100 arcseconds through a distributed micro lens array. The field of view is nearly a thousand times larger than traditional sensors, providing hardware support for the dynamic response capability of adaptive optics (Guo et al., 2024). At the same time, the programmable optical chip developed by the University of Pennsylvania has achieved optical domain training of nonlinear neural networks for the first time, maintaining accuracy while consuming less energy than traditional electronic chips, providing a new paradigm for hardware acceleration of deep neural networks (Wu et al., 2025). In addition, the real-time satellite ground quantum key distribution achieved by the team from the University of Science and Technology of China between quantum micro nano satellites and mobile ground stations has verified the high reliability of optical communication in complex channels. The composite laser communication technology used provides practical reference for the engineering application of joint solutions (Levi, 2025).

Although there has been some progress in the field of physical layer enhancement in complex channel optical communication, there are still significant shortcomings in the synergy of the technical system, as shown in Figure 2, which are reflected in structural contradictions in several dimensions. Firstly, the independent application of adaptive optical compensation technology faces an inherent contradiction between accuracy and speed. Traditional adaptive optics systems rely on the point detection principle of Shack Hartmann wavefront sensors, and their spatial sampling density is inversely proportional to the time response speed (Rockäschel & Tiziani, 2002). When increasing the number of lens arrays to improve detection accuracy, the processing delay will significantly increase, making it difficult to cope with the rapid evolution of wavefront distortion in strong turbulent environments. Research has shown that traditional Shack Hartmann wavefront sensors have a small field of view and limited ability to capture the spatiotemporal dynamic evolution of wide area turbulence in spatially non-uniform turbulent scenes (Go et al., 2024). The wavefront reconstruction algorithm is mostly based on Zernike polynomial fitting. When describing the non-Gaussian distribution characteristics of non-Kolmogorov turbulence, the fitting error will significantly increase with the decrease of turbulence coherence length. The problem of calibration efficiency attenuation is prominent in complex terrain scenes such as urban building clusters (Boreman & Dainty, 1996). In addition, the application of deep neural networks in optical signal processing faces practical bottlenecks in generalization and adaptation. The current signal compensation models based on DNN mostly adopt offline training and online inference mode, and their performance highly depends on the consistency between the statistical characteristics of the training dataset and the actual channel (Fan et al., 2020). Related experimental studies have shown that when the channel characteristics deviate from a certain range of the training sample distribution, the performance of CNN based equalizers will significantly deteriorate (Hassan et al., 2022). At the same time, the traditional von Neumann architecture of electronic chips is difficult to support the parallel computing requirements of neural networks. In high-speed optical signal processing scenarios, the problem of computational delay restricts its application (Shin & Yoo, 2019). The phase amplitude coupling characteristics of optical signals naturally do not match the real domain operations of neural networks, resulting in difficulty in achieving ideal phase noise compensation accuracy. Moreover, the research on technology integration has not yet broken through the primary stage of mechanical splicing, and the collaborative efficiency has not been fully released. The existing joint schemes mostly adopt a serial structure of “adaptive optical correction+neural network equalization”, lacking a dynamic feedback mechanism (Murphy et al., 2024). The correction residuals of the adaptive optics module did not participate in the parameter updates of the neural network, making it difficult for the network to adapt to the time-varying characteristics of the channel (Murphy et al., 2024). The AO-DNN architecture has been shown in experiments to have a significant decrease in network convergence speed in turbulent intensity mutation scenarios, and requires retraining to restore performance. More importantly, there is a spatiotemporal mismatch between the hardware platform for wavefront detection and signal processing. The lack of synchronization mechanism between the spatial resolution of adaptive optics system and the time sampling rate of neural network results in additional frame alignment errors during data interaction, which can have adverse effects on signal phase in coherent optical communication.

Figure 2 Insufficient Current Research

Existing research lacks sufficient understanding of the multidimensional coupling characteristics of complex channels, focusing mainly on compensating for single distortion factors while neglecting the synergistic mechanisms of atmospheric turbulence, fiber nonlinearity, and platform vibration. Experimental verification shows that when multiple distortions coexist, their combined effect is not simply a superposition, but will generate new distortion patterns, and the compensation efficiency of traditional single techniques will degrade. This cognitive limitation poses fundamental obstacles to the engineering application of existing solutions in complex environments, and there is an urgent need to construct a new fusion framework to overcome technological bottlenecks.

Based on this, this article focuses on the issue of enhancing the reliability of the physical layer of optical communication in complex channels. In response to the challenges of signal distortion caused by atmospheric turbulence, fiber multimode coupling, etc., a DNN-AO fusion architecture is constructed to integrate spatial, temporal, and polarization domain information through tensor quantification representation. With the help of dual stream attention mechanism, efficient cross domain information processing is achieved. Combined with the parallel pipeline and ping-pong buffer design of FPGA, DSP, and Edge TPU heterogeneous platforms, the contradiction between traditional technology in accuracy and real-time performance, generalization ability, and hardware coordination is solved; The experiment shows that this scheme is significantly better than SPGD and pure DNN schemes in nine indicators such as RMS, PVE, and Q factor. Its theoretical value lies in the construction of a new paradigm for complex channel collaborative compensation that combines physical priors with data fitting. Technologically, it breaks through the engineering contradiction between accuracy and microsecond level response, and provides key support for emerging scenarios such as 6G ultra wideband transmission and air space integrated networks, promoting the leap of optical communication physical layer enhancement technology from theory to practice.

II. Theoretical Technological Evolution

As the core infrastructure supporting the efficient operation of modern information society, the physical layer performance of optical communication directly affects the capacity, speed, and reliability of information transmission. As shown in Figure 3, with the rapid development of communication technology, the channel environment faced by the physical layer of optical communication is becoming increasingly complex, and technical challenges are becoming increasingly prominent. To further explore its development trajectory, existing problems, and future trends, the following will conduct a systematic analysis of the physical layer of optical communication from multiple dimensions.

From the perspective of technological evolution, the physical layer of optical communication has undergone decades of development, gradually moving from early intensity modulation direct detection (IM-DD) systems to the era of coherent optical communication. The modulation format has been upgraded from OOK to 16QAM, 64QAM, and even higher-order constellation modulation, and the transmission rate has jumped from Mbps to Tbps. The fusion application of wavelength division multiplexing (WDM), time division multiplexing (TDM) and other technologies has enabled the single fiber transmission capacity to continuously break through the Pb/s barrier (Rahaim & Little, 2017; Saradhi & Subramaniam, 2009; Esmail & Fathalah, 2012). In this evolution process, the physical layer always plays a key role as a link between the previous and the next, supporting the service quality guarantee of the network layer upwards and adapting to the physical characteristics of the transmission medium downwards. Its technological breakthroughs often become the core driving force for upgrading the entire communication system.

From the perspective of transmission media, the physical layer of optical communication covers two major systems: fiber optic communication and free space optical communication. Although both belong to the category of optical transmission, they face completely different technological challenges. Fiber optic communication, with its advantages of low loss and high bandwidth, has become the mainstream choice for terrestrial backbone networks. However, its physical layer performance is limited by the inherent linear and nonlinear effects of optical fibers, including chromatic dispersion and polarization mode dispersion, which can cause time delay and inter symbol interference in the transmission of optical signals with different frequencies and polarization states (Guan et al., 2018); Nonlinear effects such as Kerr effect and stimulated Raman scattering accumulate with the increase of transmission power and distance, leading to signal distortion and spectral expansion. Especially in ultra high speed and long-distance transmission scenarios, the coupling effect of these effects increases the difficulty of signal processing in the physical layer (Soman, 2021). Free space optical communication, with its flexible deployment and high bandwidth potential, has become an important supplement to satellite communication, emergency communication, and other scenarios. However, its physical layer is directly exposed to complex environments such as atmospheric channels and platform vibrations. The refractive index fluctuations caused by atmospheric turbulence can lead to beam drift, intensity scintillation, and wavefront distortion, while rain and fog attenuation can significantly reduce the received optical power. Under adverse weather conditions, it may even cause link interruption. The randomness of these channel characteristics places strict requirements on the real-time compensation capability of the physical layer (Chan, 2007; Lopez Martinez et al., 2015).

From the perspective of application scenario expansion, the service boundary of the physical layer of optical communication is constantly extending, from traditional telecommunications backbone networks to emerging fields such as data center interconnection (DCI), quantum communication, and deep-sea communication. The demand for physical layer performance in different scenarios presents a complex superposition (Paraschis & Raj, 2020). In the scenario of data center interconnection, the demand for short distance high bandwidth drives the evolution of the physical layer towards low power consumption and miniaturization, and the cost and power consumption issues of coherent technology become limiting factors; In the quantum key distribution chain, the physical layer needs to ensure both the fidelity transmission of quantum states and the efficient transmission of classical signals, which imposes almost stringent requirements on the phase stability and noise suppression of optical signals (Hugues Salas et al., 2019); In the integrated network of space, sky, and earth, the physical layer needs to adapt to the diverse channel environment from ground fiber to near earth space and deep space, and cope with the time-varying characteristics of channels from milliseconds to seconds. This cross scenario channel complexity makes it difficult for a single technology to meet the physical layer enhancement requirements of all scenarios.

A thorough analysis of the technical composition of the physical layer reveals that its performance enhancement relies on the synergistic effect of transmission medium optimization, signal modulation innovation, and distortion compensation enhancement. In terms of transmission media, new optical fibers suppress dispersion by changing the refractive index distribution, while free space optical communication adjusts the beam propagation characteristics through adaptive optical elements. However, the physical limitations of the medium itself determine its limited optimization space (Ramaswami, 2002). In terms of signal modulation, every innovation in modulation technology from binary modulation to high-order modulation, from intensity modulation to coherent modulation, has brought about a leap in spectral efficiency. However, high-order modulation signals are more sensitive to noise and distortion, significantly increasing the dependence of the physical layer on channel stability (He et al., 2022). Distortion compensation, as the core means of physical layer enhancement, has undergone an evolution from analog compensation to digital compensation. Early analog compensation achieved coarse-grained adjustment through optical domain devices, while the introduction of digital signal processing technology greatly improved compensation accuracy. Digital compensation schemes based on adaptive equalization, Kalman filtering, and other algorithms have become the mainstream choice for suppressing linear distortion.

However, with the exponential growth of channel complexity, traditional compensation techniques are facing severe challenges (Ma, 2023). In fiber optic communication, when the transmission rate exceeds 400Gbps and the distance exceeds 1000km, the coupling effect of nonlinear effects and dispersion exhibits strong nonlinear and non-stationary characteristics. Traditional DSP algorithms based on linear models are difficult to achieve accurate compensation, resulting in difficulty in controlling the signal error rate below the communication level requirement of 10^-12. In free space optical communication, the spatiotemporal evolution characteristics of atmospheric turbulence dynamically change with altitude and meteorological conditions. The correction speed of traditional adaptive optics systems is difficult to match the microsecond level changes of turbulence, especially under strong turbulence conditions. Correction residuals can cause a decrease of more than 10dB in the signal-to-noise ratio of the received signal, seriously affecting communication quality. At the same time, emerging businesses have increasingly stringent real-time requirements for the physical layer. The 1ms end-to-end latency target proposed in the 6G vision forces the physical layer signal processing delay to be controlled at the microsecond level, which is in sharp contradiction with the high computational complexity of traditional digital compensation algorithms. How to achieve high-precision and low latency physical layer enhancement in complex channel environments has become a core issue that urgently needs to be broken through in the field of optical communication.

Behind this technological dilemma lies a triple contradiction between understanding distortion mechanisms in complex channels, adapting compensation techniques, and ensuring real-time performance. On the one hand, the multi physics coupling characteristics of channel distortion have not been fully understood, resulting in limitations in the physical basis of compensation models; On the other hand, existing compensation technologies are mostly designed for single distortion types and lack collaborative processing capabilities for composite distortions. For example, adaptive optics is good at handling spatial domain distortions, while DSP algorithms focus on temporal and frequency domain distortions. The technical barriers between the two make it difficult to unleash collaborative efficiency; More importantly, there is a lack of collaborative design between physical layer hardware and algorithms. There is a computing power gap between the computing power of traditional electronic chips and the high-speed transmission of optical signals, and the flexibility of optical domain signal processing is not as good as that of the digital domain. This mismatch between hardware and algorithm adaptation has become a bottleneck that constrains real-time enhancement of the physical layer.

In this context, the enhancement technology of the physical layer of optical communication is evolving towards interdisciplinary integration. The deep integration of optical engineering, information theory, and computer science provides new possibilities for breaking through technological bottlenecks. The combination of adaptive optics technology and deep neural network is a typical embodiment of this fusion trend. The former relies on the wavefront control principle of optical engineering to achieve rapid correction of spatial distortion, and the latter relies on the nonlinear fitting ability of computer science to achieve accurate modeling of complex distortion. The cooperation of the two is expected to build a comprehensive compensation system covering multiple dimensions of space, time and frequency, opening up a new path for the performance leap of the physical layer of optical communication in complex channels.

Figure 3 Development of Optical Communication Technology

III. Joint Compensation Framework

The distortion of optical signals in complex channels exhibits multidimensional coupling characteristics, and traditional single domain modeling methods are difficult to capture their spatiotemporal dynamic evolution laws. As shown in Figure 4, the joint compensation framework proposed in this paper is based on Zhang quantization representation, and through the DNN-AO architecture, combined with real-time processing of dual stream attention mechanism, achieves accurate modeling of time-varying channels and efficient distillation of cross domain information. This framework breaks through the dual limitations of physical model simplification and data-driven generalization in traditional compensation techniques, and constructs a full chain solution from distortion representation to real-time processing.

Figure 4 Schematic diagram of joint compensation framework

1. Zhang quantization characterization of complex channel distortion

The Zhang quantization characterization of complex channel distortion aims to integrate multidimensional information from spatial, temporal, and polarization domains through high-dimensional tensor structures, achieving a complete characterization of distortion characteristics. This representation method preserves the interpretability of the physical model and provides structured input for data-driven compensation algorithms.

The wavefront phase distortion caused by atmospheric turbulence can be deconstructed as an orthogonal projection of an infinite dimensional Hilbert space, which is essentially the dynamic evolution of the Zernike coefficient tensor Z∈R^N×K in the spatiotemporal domain, where N is the mode order and K is the time frame index. The traditional static Kolmogorov spectral model fails due to ignoring memory effects (Yousefi, 2016). This paper introduces a non-stationary Markov transition kernel, expressed as follows:

（1）

In the equation, Γ=diag((n+1)^−3/2) represents the Friedel parameter r_0​ constraint on the turbulent power spectrum, and Φ_t​ is the time-varying state transition matrix.

The Markov evolution chain modeling of Zernike polynomial coefficients is aimed at the wavefront distortion caused by atmospheric turbulence and platform vibration. It decomposes it into orthogonal Zernike polynomial basis function combinations and captures the temporal correlation of distortion through the Markov properties of coefficient sequences. Due to the state continuity of wavefront distortion in time-varying channels in a short period of time, the Zernike coefficients at adjacent moments satisfy the assumption of no aftereffect. Therefore, the dynamic transition probability of the coefficients can be described by a Markov chain to achieve probabilistic prediction of the distortion state at future moments. This modeling method transforms wavefront distortion in the spatial domain into a temporal evolution process with low dimensional coefficients, significantly reducing the computational complexity of dynamic compensation.

The polarization degradation caused by underwater scattering and multi-mode coupling is essentially a distortion of the differential manifold of the Stokes vector =(S0​,S1​,S2​,S3​)^⊤ on the Poincaré sphere. This article constructs a Lie algebra representation of polarization distortion.

Firstly, perform differential geometry modeling and define the polarization transfer tensor:

（2）

Satisfy:

（3）

Where Λ(z) is the unitary irreducible representation of depolarization noise, dep is the anisotropic scattering noise base.

Next, compact extraction of statistical invariants is performed, using Cartan decomposition J_p​=Ω_rot​+D_diat​ to separate the rotational component and the dichroic component.

The polarization perturbation transfer function in Stokes vector space focuses on the polarization state distortion in optical fibers and atmospheric channels. By constructing the mathematical space of polarization states through the four components (S0-S3) of Stokes vectors, a perturbation transfer function is established to describe the variation of polarization states with transmission distance. In multimode optical fibers or strong turbulent environments, the polarization state of optical signals undergoes random rotation and depolarization effects. The transfer function integrates the spatial distribution of parameters such as azimuth and ellipticity in tensor form, achieving quantitative characterization of polarization perturbation anisotropy. Tensioning processing enables polarization distortion on different transmission paths to be superimposed and decomposed through tensor operation, providing a unified mathematical basis for cross domain compensation.

2. Physical constraint embedding architecture of DNN-AO

The physical constraint embedding architecture of DNN-AO integrates the hardware characteristics and physical laws of adaptive optics systems into the structural design of neural networks, avoiding the common problem of “physical inconsistency” in data-driven models and ensuring that the compensation effect conforms to the actual working principle of optical systems.

The cyclic consistency loss function of wavefront sensing residual feedback aims to reduce the deviation between the wavefront distortion predicted by the neural network and the actual detection residual of the adaptive optics system. The cyclic consistency loss function of residual feedback in wavefront sensing is based on the wavefront reconstruction error optimization model of the Shack Hartmann sensor, and the physical domain constraint expression is as follows:

（4）

Among them, A∈R^2m×n is the matrix of the influence function of the deformable mirror, and Ω=diag(1/(v_max−v_min)) is the regularization term of the actuator stroke boundary.

The expression for data domain cyclic conservation is as follows:

（5）

Among them, W_d^† represents the pseudo inverse operator of Zernike mode to slope space.

The expression for the H¹ Soberlev norm is as follows:

（6）

The traditional loss function only focuses on the error between the output and the label, while the cyclic consistency loss constructs a “prediction correction feedback” loop, requiring the compensation generated by the network to be consistent with the theoretical prediction residual after being executed by adaptive optics hardware. This constraint introduces the actual response characteristics of the physical layer hardware into the loss calculation, forcing the network to learn compensation strategies that fit the hardware performance and reduce systematic errors caused by model hardware mismatch.

The convolution response characteristics of controllable deformable mirrors are mapped by simulating the spatial response function of deformable mirrors through neural network layers, achieving accurate modeling of their physical behavior. The convolution response characteristic mapping of controllable deformable mirrors is designed using a neural network driven by an elastic constitutive model. The expression of the mirror displacement field control equation is as follows:

（7-8）

​​The expression for the physical constraint CNN design criteria is as follows:

(9-11)

Each driving unit of the deformable mirror exhibits spatial coupling effects, where the displacement of a single unit affects the wavefront shape of adjacent regions. This characteristic can be characterized by the spatial weight distribution of the convolution kernel. Embedding this mapping into the convolutional layer of the network enables the neural network to naturally follow the physical constraints of the deformable mirror when generating compensation instructions, avoiding outputting invalid instructions beyond the hardware drive range and improving the executable performance of compensation instructions.

The phase unwrapping regularization method based on Lie group structure addresses the periodic ambiguity problem of optical signal phase, utilizing the algebraic structure of Lie groups to constrain the phase unwrapping process of neural networks.

The phase unwrapping regularization method based on Lie group structure adopts a phase optimization system under the framework of differential geometry. The expression for unitary group homomorphic mapping is as follows:

（12）

The constraint expression of Riemannian manifold is as follows:

（13）

The expression for the iterative unwrapping operator is as follows:

（14）

In strong distortion scenarios, traditional phase unwrapping algorithms are prone to cumulative errors due to noise interference, while the Lie group structure ensures global consistency in unwrapping results by defining group operation rules for phase rotation, avoiding local error diffusion. This regularization method transforms geometric constraints in topology into regularization terms for network training, ensuring that the phase compensation results conform to both mathematical consistency and the physical properties of optical phase.

3. Real time dual stream attention mechanism

The dual stream attention mechanism for real-time processing separates and prioritizes key distortion components that significantly affect communication performance, greatly improving processing speed while ensuring compensation accuracy, and meeting the real-time requirements of high-speed optical communication. The dual stream structure designs dedicated processing paths for the distortion characteristics in both spatial and frequency domains, while the attention mechanism dynamically allocates computing resources and focuses on the dominant distortion factors under the current channel conditions.

The decoupling distillation of spatial wavefront distortion and frequency domain symbol interference is achieved through a dual channel network structure to separate and fuse the two types of distortions. The spatial channel uses multi-scale convolution to extract spatial features of wavefront distortion, with a focus on handling beam phase fluctuations caused by atmospheric turbulence and deformable mirror residuals; The frequency domain channel analyzes the signal spectrum shift through Fourier transform and extracts features for inter symbol interference caused by nonlinearity. The decoupling distillation process suppresses cross interference between spatial and frequency domains through dynamic adjustment of attention weights. When turbulence dominates distortion, the weight of the spatial channel is increased to enhance wavefront correction; When dispersion becomes the main limiting factor, the proportion of resources in the frequency domain channel increases to ensure efficient compensation for symbol interference.

The timing prediction truncation strategy of Gated Recurrent Unit (GRU) optimizes the long sequence prediction process for real-time requirements by dynamically truncating redundant historical data to reduce computational latency.

The lightweight GRU unit with physical constraints designed in the timing prediction truncation strategy of GRU is expressed as follows:

（15）

GRU networks typically rely on longer historical sequences to improve accuracy in temporal prediction, but excessively long sequences can lead to increased processing latency. The truncation strategy adaptively adjusts the length of the input sequence based on the rate of change of the current channel: during the stationary phase of the channel, shorter sequences are used to reduce computational complexity; When significant distortion changes are detected, the sequence length is automatically extended to ensure prediction accuracy. At the same time, the gating mechanism enhances the weight of recent keyframes and preserves information that significantly affects future states in the truncated sequence, achieving a balance between accuracy and real-time performance. This strategy keeps the prediction delay of GRU at the microsecond level, meeting the strict requirements of 6G communication for physical layer processing speed.

IV. Hardware Implementation

As shown in Figure 5, the core of this article revolves around the design of hardware collaboration for three major modules: adaptive optical compensation, neural processing acceleration, and delay constraint collaboration.

1. Low latency reconstruction of adaptive optical compensation link

The FPGA based parallel wavefront slope calculation unit adopts Xilinx Virtex UltraScale+VU19P FPGA to construct a parallel processing architecture, integrating 32 dedicated computing units, each configured with an 8-bit data path, and achieving microsecond level response for wavefront slope calculation through pipeline technology. Based on the physical model of the Shack Hartmann wavefront sensor, a global spot adaptive matching algorithm is adopted to improve the dynamic range of traditional local matching methods. The FPGA internally implements Zernike polynomial expansion and Hausdorff distance optimization through hardware description language, which can maintain measurement accuracy even in scenarios where some light spots are missing. This design uses distributed BRAM to store real-time wavefront data, combined with high-speed LVDS interface and external analog-to-digital converter to meet the real-time detection requirements in strong disturbance environments such as atmospheric turbulence.

The PWM carrier synchronization mechanism of piezoelectric ceramic drivers is based on the Texas Instruments LMX2594 phase-locked loop to construct a high-precision clock source. The carrier frequency is dynamically configured through the SPI interface, supporting ± 0.1 ° phase adjustment accuracy. Using F280049C DSP to achieve multi-channel PWM synchronization, triggering the TBCTR register reset of all ePWM modules through synchronization events to ensure that the phase deviation of 32 drive signals is less than 0.1 degrees. Design a feedforward compensator based on recursive least squares to address the hysteresis characteristics of piezoelectric ceramics. The compensator outputs a compensation voltage through a 16 bit DAC and suppresses nonlinear distortion to low values within a 10kHz bandwidth, meeting the closed-loop control requirements of adaptive optics systems.

2. Hardware friendly deployment of neural processing engines

The phase recovery network adopts a symmetric quantization strategy to limit the neural network weights and activation values within the range of -128~127. Combined with a compressed sensing based phase recovery algorithm, it reduces the model storage while maintaining a small PSNR loss. Using TensorFlow Lite for quantization aware training and introducing KL divergence loss function to optimize quantization error, achieving high phase recovery accuracy on the NUFFT dataset. When deploying hardware, choose Google Edge TPU as the inference accelerator, whose 8-bit fixed-point computing unit supports 2TOPS computing power.

Further build a heterogeneous computing platform consisting of FPGA (Xilinx Zynq UltraScale+) and DSP (TI TMS320C6678), using AXIDMA controller to achieve data pre reading and computation overlap. By using ping-pong buffer mechanism to improve the parallelism of data handling and convolution operations, combined with NHWC data layout optimization of cache utilization, the sustained computing power of the neural processing engine reaches 1.8 TOPS.

3. Analysis of Time Delay Coupling Constraints in Joint Systems

This article uses Trimble Thunderbolt 6100 atomic clock to provide a 10MHz reference clock, generates a 1GHz system clock through the MMCM module of FPGA, and combines the clock data recovery technology of PCIe 5.0 protocol to control the time deviation of cross module data interaction within 125ps. Design a frame marking mechanism based on Gray code, achieve bit synchronization of data stream through 8B/10B encoding, and maintain data integrity. In terms of the Lyapunov stability criterion for the feedback loop, a state space model is established that includes time delay τ and system gain K. The Lyapunov matrix P is determined by solving the Riccati equation. Matlab/Simulink is used for simulation verification, and the system convergence time is low in the scenario of sudden turbulence intensity changes.

Figure 5 Hardware Implementation Diagram

V. Empirical Test

1. Experimental environment

The experimental environment of this article is shown in Table 1.

Table 1 Experimental Environment

Equipment Category	Model/Design Scheme
Core Processing Equipment
Main Processing FPGA	Xilinx Virtex UltraScale+ VU19P
Drive Control DSP	TI TMS320F280049C
Neural Accelerator	Google Coral Dev Board Edge TPU
Heterogeneous Computing Bridge	Xilinx Zynq UltraScale+ MPSoC
Optical & Channel Equipment
Light Source System	Narrow-linewidth Laser (Self-developed)
Shack-Hartmann Wavefront Sensor	Thorlabs WFS150-7AR
Piezoelectric Deformable Mirror	Boston Micromachines Kilo-DM
Atmospheric Turbulence Simulator	Self-developed Turbulence Pool (Heating Wire Array)
Fiber Nonlinearity Simulator	Keysight N7768C Optical Vector Analyzer
Testing & Calibration Equipment
High-Speed Oscilloscope	Keysight DSOZ634A
Laser Interferometer	Zygo GPI-XP
Optical Power Meter	Thorlabs PM100D
Spectrum Analyzer	Keysight N9030B
Clock Synchronization Source	Trimble Thunderbolt 6100
Auxiliary Equipment
High-Precision DC Power Supply	Keysight N6951A
Temperature Chamber	Thermotron SE-150
Optical Attenuator	Thorlabs NDL-50C-4
Interference Calibrator	Zygo Verifire XP-D
Data Acquisition Card	NI PXIe-6368

2. Comparison plan

This article selects the classic SPGD scheme and the pure data-driven DNN scheme as comparative schemes.

The classic SPGD scheme, as a traditional correction method in the field of adaptive optics, achieves wavefront optimization by applying random perturbations to control parameters and estimating gradients based on evaluation functions. Its core relies on the stochastic parallel gradient descent algorithm to complete closed-loop iteration. This scheme has a simple structure and does not require an accurate wavefront sensing model. It has stable correction effects in low turbulence intensity scenarios, but is limited by the statistical characteristics of gradient estimation. The convergence speed decreases linearly with the increase of disturbance dimension, and it is prone to local optima in nonlinear distortion scenarios caused by strong turbulence, making it difficult to meet microsecond level real-time response requirements.

The pure data-driven DNN scheme relies on large-scale annotated data to train deep neural networks, and directly outputs wavefront compensation signals through end-to-end mapping, eliminating the physical model derivation step in traditional algorithms. This scheme utilizes GPU or dedicated neural accelerator to achieve parallel computing within milliseconds, exhibiting high compensation accuracy in typical scenarios covered by training data. However, due to the lack of physical constraint embedding, the generalization ability significantly decreases when the channel distortion pattern exceeds the training distribution. Moreover, model training relies on a large number of annotated samples, which can easily lead to overfitting in complex channels with dynamic changes.

3. Evaluation indicators

The main evaluation indicators selected in this article are shown in Table 2.

Table 2 Evaluation Indicators

​​Metric Name​​	​​Symbol​​	​​Technical Definition​​
​​Wavefront Residual RMS​​	RMS	Root-mean-square deviation between corrected phase front and ideal plane (units: λ)
​​Peak-to-Valley Error​​	PVE	Optical path difference between highest and lowest points on wavefront distortion surface (units: λ)
​​Strehl Ratio​​	Sr	Ratio of actual focal spot peak intensity to diffraction-limited intensity
​​Polarization Maintenance​​	DOP	Average fidelity of Stokes vectors on Poincaré sphere
​​Q-Factor	Q	Eye opening / noise standard deviation
​​Channel Capacity Gain​​	CCG	Shannon capacity improvement (dB)
​​System Latency​​	SL	End-to-end delay in microseconds
​​Power Efficiency​​	PET	Energy consumption per transmitted bit (nJ/bit)
​​Turbulence Dynamic Range​​	TDR	Maximum-to-minimum turbulence intensity ratio (dB) maintaining Sr > 0.6

4. Experimental results

The experimental results are shown in Table 3. The joint optimization scheme proposed in this paper exhibits significant advantages in various evaluation indicators, and its performance improvement is clearly related to the embedding of physical constraints and hardware architecture optimization in the scheme design. From the perspective of wavefront correction accuracy, the RMS of this scheme is only 36.3% of the traditional SPGD scheme and 57.8% of the pure DNN scheme. The PVE is reduced to 37.2% of the traditional scheme and 49.7% of the pure DNN scheme, respectively. This is directly related to the Zernike polynomial Markov chain modeling and Lie group phase unwrapping regularization in the scheme. By physically constraining the neural network output, high-frequency noise in wavefront reconstruction is effectively suppressed, resulting in a 13.6% improvement in Sr compared to the pure DNN scheme, approaching the ideal correction state.

In terms of signal transmission quality, the DOP of our proposed scheme is 7.7% higher than that of the pure DNN scheme, which confirms the effectiveness of the Stokes vector transfer function in the polarization preserving mechanism; The Q factor has increased by 86.6% compared to the traditional SPGD scheme, and CCG has achieved an absolute gain of 8.3 units. This is due to the synergistic compensation of channel nonlinear distortion by the physically constrained DNN-AO architecture, which retains the fitting ability of pure DNN for complex modes and avoids signal distortion caused by overfitting through wavefront sensing residual feedback loss function.

The correlation between system response characteristics and hardware design is particularly significant. The SL proposed in this paper is 73.6% lower than the pure DNN scheme, achieving microsecond level real-time standards. This is directly related to the 32 channel dedicated computing unit of the FPGA parallel processing unit and the DMA dual buffer pipeline design. The data pre reading and computation overlap mechanism significantly shortens the idle time of the link; PET reduces storage access energy consumption by 62.1% compared to pure DNN schemes, and even outperforms traditional SPGD schemes, thanks to the 8-bit fixed-point network compression that reduces storage access energy consumption and dynamic power management of heterogeneous architectures.

In terms of environmental adaptability, the TDRs of our proposed scheme are 40 times that of the traditional SPGD scheme and 1.6 times that of the pure DNN scheme, respectively, verifying the effectiveness of the Lyapunov stability criterion in strong turbulence scenarios. When the turbulence intensity approaches 8e-13, traditional SPGD experiences a sudden increase in RMS due to the failure of gradient estimation, while the pure DNN scheme experiences a decline in correction accuracy due to insufficient extrapolation ability of the training distribution. However, this paper’s scheme updates the physical model parameters in real time through a quantization channel representation, maintaining stable compensation performance and demonstrating the core advantage of the fusion of physical modeling and data-driven approaches.

Table 3 Experimental Results

​​Evaluation indicators​​	​​Traditional SPGD scheme​​	Pure DNN scheme	​​This article’s proposal
RMS	0.102	0.064	​​0.037​​
PVE	2.31	1.73	​​0.86​​
Sr	0.63	0.81	​​0.92​​
DOP	0.82	0.91	​​0.98​​
Q	8.2	10.7	​​15.3​​
CCG	28.6	35.2	​​43.5​​
SL	83±14.2	28±7.5	​​7.4±0.86​​
PET	5.2	8.7	​​3.3​​
TDR	<2e-14	<5e-13	​​<8e-13​​

As shown in Figure 6, the proposed joint optimization scheme achieves a global breakthrough in adaptive optical compensation performance through deep coupling between physical mechanisms and data-driven approaches. Its core value is not only reflected in the absolute advantage of individual indicators, but also in solving the inherent contradiction between accuracy and real-time, adaptability and energy consumption in traditional methods. In the strong turbulence scenario where TDR is extended to 8e-13, it can still maintain a DOP of 0.98 and a PET of 3.3 synchronously.

Figure 6 Experimental results

VI. Conclusion

Against the backdrop of increasingly prominent complex channel effects such as atmospheric turbulence and fiber optic multimode coupling, this study focuses on the core demand of enhancing the reliability of the physical layer in optical communication, and systematically explores the technical path of integrating DNN and AO compensation. By deeply analyzing the structural contradictions existing in traditional technologies, such as the inherent conflict between accuracy and speed faced by adaptive optics, the generalization bottleneck of pure data-driven DNNs, and the insufficient collaborative efficiency of early fusion schemes, a DNN-AO fusion architecture embedded with high-order physical constraints was constructed to achieve efficient integration of Zhang quantized channel representation, dual stream attention mechanism, and heterogeneous hardware collaboration.

Empirical tests have shown that the fusion framework achieves breakthrough improvements in nine key indicators, including wavefront residual RMS, PVE, Sr, DOP, and Q factor. Compared with traditional SPGD schemes and pure DNN schemes, it not only breaks through the theoretical constraints of physical modeling and data-driven separation, but also achieves high-precision engineering balance.

At the theoretical level, this study reveals that enhancing the physical layer of optical communication in complex channels is not achievable by optimizing a single technology, and a system framework that integrates multidimensional information and multi field mechanisms must be established. The Zhang quantization representation of complex channel distortion integrates spatial, temporal, and polarization domain information, and the Markov chain modeling of Zernike coefficients preserves the interpretability of physical models while providing structured inputs for data-driven algorithms. This indicates that the organic integration of physical prior knowledge and data-driven methods is the key to breaking through the bottleneck of traditional technology generalization. Future theoretical research should further deepen the exploration of multi physics coupling mechanisms, focus on the nonlinear and non-stationary characteristics of complex channels, and construct more refined dynamic evolution models. At the same time, it is necessary to strengthen the theoretical research on the integration of “physics+data”, explore the interaction mechanism between physical constraints and neural network learning, and provide more solid theoretical support for cross domain collaborative compensation.

From the perspective of technological evolution, the physical constraint embedding strategy proposed in this study, such as the cyclic consistency loss function of wavefront sensing residual feedback and the phase unwrapping regularization based on Lie group structure, effectively solves the problems of insufficient physical consistency and data-driven overfitting in traditional technologies. The collaborative design of FPGA parallel processing units and heterogeneous hardware has achieved microsecond level system latency and high energy efficiency ratio, confirming the importance of hardware architecture and algorithm design optimization. This suggests that future technology research and development should focus on building a collaborative innovation system for algorithms and hardware, continuously optimizing the integration of physical constraints and neural networks, and exploring more efficient paths for embedding physical priors; Strengthen the research and development of dedicated hardware acceleration architecture, design a dynamic scheduling mechanism for heterogeneous computing units to meet the energy efficiency requirements of optical communication scenarios, and unleash the performance potential of fusion technology.

Reference

• Kaushal, H., & Kaddoum, G. (2016). Optical communication in space: Challenges and mitigation techniques. IEEE communications surveys & tutorials, 19(1), 57-96.
• Agrell, E., Karlsson, M., Poletti, F., Namiki, S., Chen, X., Rusch, L. A., … & Uysal, M. (2024). Roadmap on optical communications. Journal of Optics, 26(9), 093001.
• Raj, A. A. B., Krishnan, P., Darusalam, U., Kaddoum, G., Ghassemlooy, Z., Abadi, M. M., … & Ijaz, M. (2023). A review–unguided optical communications: Developments, technology evolution, and challenges. Electronics, 12(8), 1922.
• Guo, Y., Hao, Y., Wan, S., Zhang, H., Zhu, L., Zhang, Y., … & Fang, L. (2024). Direct observation of atmospheric turbulence with a video-rate wide-field wavefront sensor. Nature Photonics, 18(9), 935-943.
• Wu, T., Li, Y., Ge, L., & Feng, L. (2025). Field-programmable photonic nonlinearity. Nature Photonics, 1-8.
• Levi, F. (2025). A practical leap towards secure quantum communication over long distances. NATURE.
• Rocktäschel, M., & Tiziani, H. J. (2002). Limitations of the Shack–Hartmann sensor for testing optical aspherics. Optics & Laser Technology, 34(8), 631-637.
• Go, G. H., Lee, D. G., Oh, J., Song, G., Lee, D., & Jang, M. (2024). Meta Shack–Hartmann wavefront sensor with large sampling density and large angular field of view: phase imaging of complex objects. Light: Science & Applications, 13(1), 187.
• Boreman, G. D., & Dainty, C. (1996). Zernike expansions for non-Kolmogorov turbulence. Journal of the Optical Society of America A, 13(3), 517-522.
• Fan, Q., Zhou, G., Gui, T., Lu, C., & Lau, A. P. T. (2020). Advancing theoretical understanding and practical performance of signal processing for nonlinear optical communications through machine learning. Nature Communications, 11(1), 3694.
• Hassan, S., Tariq, N., Naqvi, R. A., Rehman, A. U., & Kaabar, M. K. (2022). Performance Evaluation of Machine Learning‐Based Channel Equalization Techniques: New Trends and Challenges. Journal of Sensors, 2022(1), 2053086.
• Shin, D., & Yoo, H. J. (2019). The heterogeneous deep neural network processor with a non-von Neumann architecture. Proceedings of the IEEE, 108(8), 1245-1260.
• Murphy, S. L., Townsend, P. D., & Antony, C. (2024). SkipNet: an adaptive neural network equalization algorithm for future passive optical networking. Journal of Optical Communications and Networking, 16(11), 1082-1092.
• Rahaim, M., & Little, T. D. (2017). Interference in IM/DD optical wireless communication networks. Journal of Optical Communications and Networking, 9(9), D51-D63.
• Saradhi, C. V., & Subramaniam, S. (2009). Physical layer impairment aware routing (PLIAR) in WDM optical networks: Issues and challenges. IEEE Communications Surveys & Tutorials, 11(4), 109-130.
• Esmail, M. A., & Fathallah, H. (2012). Physical layer monitoring techniques for TDM-passive optical networks: A survey. IEEE Communications Surveys & Tutorials, 15(2), 943-958.
• Guan, K., Cho, J., & Winzer, P. J. (2018). Physical layer security in fiber-optic MIMO-SDM systems: An overview. Optics Communications, 408, 31-41.
• Soman, S. K. O. (2021). A tutorial on fiber Kerr nonlinearity effect and its compensation in optical communication systems. journal of Optics, 23(12), 123502.
• Chan, V. W. (2007). Free-space optical communications. Journal of Lightwave technology, 24(12), 4750-4762.
• Lopez-Martinez, F. J., Gomez, G., & Garrido-Balsells, J. M. (2015). Physical-layer security in free-space optical communications. IEEE Photonics Journal, 7(2), 1-14.
• Paraschis, L., & Raj, K. (2020). Innovations in DCI transport networks. Optical Fiber Telecommunications VII, 673-718.
• Hugues-Salas, E., Ntavou, F., Gkounis, D., Kanellos, G. T., Nejabati, R., & Simeonidou, D. (2019). Monitoring and physical-layer attack mitigation in SDN-controlled quantum key distribution networks. Journal of Optical Communications and Networking, 11(2), A209-A218.
• Ramaswami, R. (2002). Optical fiber communication: from transmission to networking. IEEE Communications Magazine, 40(5), 138-147.
• He, J., Giddings, R., Xi, W., & Tang, J. (2022). DSP-based physical layer security for coherent optical communication systems. IEEE Photonics Journal, 14(5), 1-11.
• Ma, M. (2023). Signal detection of large scale MIMO physical layer space optical communication system based on genetic algorithm. Results in Physics, 54, 107055.
• Yousefi, M. I. (2016). The Kolmogorov–Zakharov model for optical fiber communication. IEEE Transactions on Information Theory, 63(1), 377-391.