All linear optical devices are mode converters.

We show that every linear optical component can be completely described as a device that converts one set of orthogonal input modes, one by one, to a matching set of orthogonal output modes. This result holds for any linear optical structure with any specific variation in space and/or time of its structure. There are therefore preferred orthogonal "mode converter" basis sets of input and output functions for describing any linear optical device, in terms of which the device can be described by a simple diagonal operator. This result should help us understand what linear optical devices we can and cannot make. As illustrations, we use this approach to derive a general expression for the alignment tolerance of an efficient mode coupler and to prove that loss-less combining of orthogonal modes is impossible.