Modulational instability and higher-order rogue waves with parameters modulation in a coupled integrable AB system via the generalized Darboux transformation.

We study higher-order rogue wave (RW) solutions of the coupled integrable dispersive AB system (also called Pedlosky system), which describes the evolution of wave-packets in a marginally stable or unstable baroclinic shear flow in geophysical fluids. We propose its continuous-wave (CW) solutions and existent conditions for their modulation instability to form the rogue waves. A new generalized N-fold Darboux transformation (DT) is proposed in terms of the Taylor series expansion for the spectral parameter in the Darboux matrix and its limit procedure and applied to the CW solutions to generate multi-rogue wave solutions of the coupled AB system, which satisfy the general compatibility condition. The dynamical behaviors of these higher-order rogue wave solutions demonstrate both strong and weak interactions by modulating parameters, in which some weak interactions can generate the abundant triangle, pentagon structures, etc. Particularly, the trajectories of motion of peaks and depressions of profiles of the first-order RWs are explicitly analyzed. The generalized DT method used in this paper can be extended to other nonlinear integrable systems. These results may be useful for understanding the corresponding rogue-wave phenomena in fluid mechanics and related fields.