Ecology shapes the evolutionary trade-off between predator avoidance and defence in coral reef butterflyfishes.

Antipredator defensive traits are thought to trade-off evolutionarily with traits that facilitate predator avoidance. However, complexity and scale have precluded tests of this prediction in many groups, including fishes. Using a macroevolutionary approach, we test this prediction in butterflyfishes, an iconic group of coral reef inhabitants with diverse social behaviours, foraging strategies and antipredator adaptations. We find that several antipredator traits have evolved adaptively, dependent primarily on foraging strategy. We identify a previously unrecognised axis of diversity in butterflyfishes where species with robust morphological defences have riskier foraging strategies and lack sociality, while species with reduced morphological defences feed in familiar territories, have adaptations for quick escapes and benefit from the vigilance provided by sociality. Furthermore, we find evidence for the constrained evolution of fin spines among species that graze solely on corals, highlighting the importance of corals, as both prey and structural refuge, in shaping fish morphology.