Gaussian mixture model-hidden Markov model based nonlinear equalizer for optical fiber transmission.

The demand for high speed data transmission has increased rapidly over the past few years, leading to the development of the data center concept. As is known, nonlinear effects in optical fiber transmission systems are becoming significant with the development of transmission speed. Since it is difficult for conventional DSP algorithms to accurately capture these nonlinear distortions, many machine learning-based equalizers have been proposed. However, previous corresponding experiments mainly focused on achieving low BER while the computational complexity is much greater. In this paper, we propose a Gaussian mixture model (GMM)-hidden Markov model (HMM) based nonlinear equalizer, which utilizes the received signals' statistical characteristics as the priori information to reduce the computational complexity. The BER performance of the GMM-HMM based equalizer has been evaluated in a PAM-4 modulated VCSEL-MMF optical interconnect link, which shows an excellent capability of mitigating nonlinear distortions. In addition, the computational complexity of GMM-HMM based equalizer is about 73% lower than that of recurrent neural networks (RNN) based methods with similar BER performance.